Methods and apparatuses for encoding/decoding high resolution images

ABSTRACT

The invention relates to methods and apparatuses for encoding/decoding high resolution images, which involve setting the size of the prediction unit to be encoded to an expanded macro-block size in accordance with the temporal frequency characteristics or spatial frequency characteristics among pictures to be encoded, and performing motion prediction motion compensation, and transformation on the basis of a set prediction unit size. In addition, the methods and the apparatuses of the present invention involve dividing a macro-block having a pixel size of 32*32 or 64*64 into at least one partition on the basis of an edge, and performing encoding processes for each partition. Accordingly, encoding efficiency can be improved for high definition (HD) or higher resolution high-resolution images.

TECHNICAL FIELD

The present invention is directed to encoding and decoding images, andmore specifically to an encoding method that may apply to highdefinition (HD) images and an encoding apparatus of performing theencoding method, a decoding method and a decoding apparatus ofperforming the decoding method.

RELATED ART

In general image compression methods, one picture is separated intomultiple blocks each having a predetermined size. Inter-prediction andinfra-prediction technologies are used to remove redundancy betweenpictures so as to increase compression efficiency.

An inter-prediction encoding method, which is represented by a motioncompensation prediction encoding method, compresses images byeliminating redundancy between pictures.

In the motion compensation prediction encoding method, a region similarto a block being currently encoded is searched from at least onereference picture positioned before or behind a currently encodedpicture to generate a motion vector (MV) which is then used for motioncompensation, thereby generating a prediction block, and a differentialvalue between the generated prediction block and the current block isthen subjected to discrete cosine transform (DCT) and quantized. Thequantized result is entropy encoded and transmitted.

Conventionally, for motion compensation prediction, macro-blocks havingvarious sizes, such as 16×16, 8×16, or 8×8 pixels, are used, and fortransform and quantization, macro-blocks of 8×8 or 4×4 pixels are used.

However, the above-mentioned existing block size is not appropriate forencoding high-resolution images having an HD-level or higher.

The block size used for general inter prediction includes 4×4, 8×8, or16×16 pixels.

Generally, among the intra and inter prediction methods, a predictionmethod with higher encoding efficiency is selected and used. Theconventional block-based prediction methods rid only one of the temporaland spatial redundancies included in an image, whose removal providesmore encoding efficiency than removal of the other. However, even thoughone of the inter and intra prediction methods is used to remove one typeof redundancy included in an image, the other type of redundancy stillremains in the image, so that it is not likely to contribute to anenhancement in encoding efficiency.

For example, the conventional block-based prediction methods would notgreatly work on images including both temporal and spatial redundancies.

Further, the above-mentioned block-based prediction methods are notappropriate for encoding HD or higher images.

Specifically, in the case of a low-resolution small image to be encoded,it may be efficient in terms of accuracy of motion prediction and bitrate to perform motion prediction and compensation with a small size ofblock. However, in the case of a high-resolution large image, when itsmotion prediction and compensation are conducted in the unit of a blockhaving a size of 16×16 or less, the number of blocks included in onepicture may be exponentially increased, thus resulting in an increase inencoding processing load and the amount of data compressed. Thus, thebit rate increases.

Further, as the resolution of an image increases, a region with nodetail or no deviation becomes broader. Accordingly, when motionprediction and compensation are conducted with a block having a size of16×16 pixels as in the conventional methods, encoding noise mayincrease.

Meanwhile, the method of encoding images using inter prediction predictspixel values based on pixel correlation between blocks from pixel valuesin the blocks surrounding a current block, which have been alreadyencoded, such as an upper block, a left block, an upper and left block,and an upper and right block in a current frame (or picture), andtransmits their prediction errors.

In the inter-prediction encoding, among several prediction directions(horizontal, vertical, diagonal, or average, etc.), an optimalprediction mode (prediction direction) is selected to be suited forcharacteristics of an image to be encoded.

In the existing H.264/AVC standards, in the case that theinter-prediction encoding applies to a 4×4 pixel unit block, among nineprediction modes (prediction modes 0 to 8), one most appropriateprediction mode is selected every 4×4 pixel blocks, and the selectedprediction mode (prediction direction) is encoded on a per-4×4 pixelblock basis. Further, in the case that the inter-prediction encodingapplies to a 16×16 pixel unit block, among four prediction modes(vertical prediction, horizontal prediction, average prediction, andplanar prediction), one most appropriate prediction mode is selectedevery 16×16 pixel blocks, and the selected prediction mode (predictiondirection) is encoded on a per-16×16 pixel block basis.

In the existing intra-prediction encoding, a predetermined number ofprediction directions are predefined only for M×M square-typesymmetrical pixel blocks (M=4, 8, or 16) to perform the intra-predictionencoding. That is, conventionally, M×M-size symmetrical partitioningonly is applied for inter-prediction encoding so that a square-typesymmetrical block is used as a basic unit for the intra-predictionencoding.

In the case of fulfilling the intra-prediction encoding using only theexisting symmetrical pixel blocks, there is a limit to raising codingefficiency, so a method to enhance coding efficiency is required. Inparticular, when it comes to high-resolution images having the highdefinition (HD) or higher resolution, the intra-prediction encodingwhich uses only the symmetrical pixel blocks has a limit as toincreasing encoding efficiency. Therefore, a need exists which canincrease encoding efficiency.

DISCLOSURE Technical Problem

A first object of the present invention is to provide an image encodingand decoding method that can increase encoding efficiency forhigh-resolution images having an HD or higher resolution.

A second object of the present invention is to provide an image encodingand decoding apparatus that can increase encoding efficiency forhigh-resolution images having an HD or higher resolution.

A third object of the present invention is to provide anintra-prediction encoding method and apparatus that can apply tohigh-resolution images having an HD or higher resolution.

A fourth object of the present invention is to provide anintra-prediction decoding method and apparatus that can apply tohigh-resolution images having an HD or higher resolution.

A fifth object of the present invention is to provide an image encodingand decoding method that can increase encoding efficiency whilemaintaining quality of high-resolution images having an HD or higherresolution.

A sixth object of the present invention is to provide an image encodingand decoding apparatus that can increase encoding efficiency whilemaintaining quality of high-resolution images having an HD or higherresolution.

Technical Solution

To achieve the first object, according to an aspect of the presentinvention, there is provided a method of encoding an image, the methodcomprising the steps of receiving at least one picture to be encoded,determining a size of a block to be encoded based on a temporalfrequency characteristics between the received at least one picture, andencoding a block having the determined size.

To achieve the first object, according to another aspect of the presentinvention, there is provided a method of encoding an image having an HDor higher resolution, the method comprising the steps of generating aprediction block by performing motion compensation on a prediction unithaving a size of N×N pixels, wherein N is a power of two and N is equalto or more than 32, obtaining a residue by comparing the prediction unitwith the prediction block, and transforming the residue. The image has aresolution which is an HD (High Definition) or higher resolution, andwherein the prediction unit has a size of an extended macro-block. Thestep of transforming the residue includes performing DCT (DiscreteCosine Transform) on the extended macro-block. The prediction unit has asize of N×N pixels, wherein N is a power of two and N is equal to ormore than 128. The size of the prediction unit may be restricted to64×64 pixels or less in consideration of complexity of an encoder anddecoder.

To achieve the first object, according to another aspect of the presentinvention, there is provided a method of encoding an image, the methodcomprising the steps of receiving at least one picture to be encoded,determining a size of a prediction unit to be encoded based on a spatialfrequency characteristics of the received at least one picture, whereinthe size of the prediction unit has a size of N×N pixels, wherein N is apower of two and N is equal to or more than 32, and encoding theprediction unit having the determined size.

To achieve the first object, according to another aspect of the presentinvention, there is provided a method of encoding an image, the methodcomprising the steps of receiving an extended macro-block having a sizeof N×N pixels, wherein N is a power of two and N is equal to or morethan 32, detecting a pixel belonging to an edge among blocks adjacent tothe received extended macro-block, splitting the extended macro-blockinto at least one partition based on the pixel belonging to the edge,and performing encoding on a predetermined partition of the split atleast one partition.

To achieve the first object, according to an aspect of the presentinvention, there is provided a method of decoding an image having a HDor higher resolution, the method comprising the steps of receiving anencoded bit stream, obtaining size information of a prediction unit tobe decoded from the received bit stream, wherein a size of theprediction unit is N×N pixels, wherein N is a power of two and N isequal to or more than 32, inverse-quantizing and inverse-transformingthe received bit stream to obtain a residue, performing motioncompensation on a prediction unit having a size corresponding to theobtained size information of the prediction unit to generate aprediction block, and adding the generated prediction block to theresidue to restore the image. The prediction unit has a size of anextended macro-block. The step of transforming the residue includesperforming inverse DCT (Discrete Cosine Transform) on the extendedmacro-block. The prediction unit has a size of N×N pixels, wherein N isa power of two, and wherein the size of the prediction unit isrestricted to 64×64 pixels or less in consideration of complexity of anencoder and a decoder. The prediction unit corresponds to a leaf codingunit when a coding unit having a variable size is hierarchically splitand reaches a maximum permissible level or level depth. The methodfurther comprises the step of obtaining partition information of theprediction unit to be decoded from the received bit stream. The step ofperforming motion compensation on a prediction unit having a sizecorresponding to the obtained size information of the prediction unit togenerate a prediction block includes splitting the prediction unit intopartitions based on the partition information of the prediction unit toperform the motion compensation on the split partitions. The partitionsplitting is achieved by an asymmetric partitioning method. Thepartition splitting is achieved by a geometrical partitioning methodwith a shape other than a square shape. The partition splitting isachieved by a partitioning method along a direction of an edge. Thepartitioning method along the edge direction includes the steps ofdetecting a pixel belonging to the edge among blocks adjacent to theprediction unit and splitting the prediction unit into at least onepartition based on the pixel belonging to the detected edge. Thepartitioning method along the edge direction applies to intraprediction.

To achieve the first object, according to another aspect of the presentinvention, there is provided a method of decoding an image, the methodcomprising the steps of receiving an encoded bit stream, obtaining sizeinformation and partition information of a macro-block to be decodedfrom the received bit stream, obtaining a residue by inverse-quantizingand inverse-transforming the received bit stream, splitting an extendedmacro-block having a size of 32×32, 64×64, and 128×128 pixels based onthe obtained size information and partition information of themacro-block into at least one partition, performing motion compensationon a predetermined partition of the split at least one partition togenerate a prediction partition, and adding the generated predictionpartition to the residue thereby restoring the image.

To achieve the second object, according to an aspect of the presentinvention, there is provided an apparatus of encoding an imagecomprising a prediction unit determining unit configured to receive atleast one picture to be encoded and to determine a size of a predictionunit to be encoded based on a temporal frequency characteristics betweenthe received at least one picture and a spatial frequencycharacteristics of the received at least one picture and an encoderconfigured to encode the prediction unit having the determined size.

To achieve the second object, according to another aspect of the presentinvention, there is provided an apparatus of encoding an imagecomprising an entropy decoding unit configured to decode a received bitstream to generate header information, a motion compensating unitconfigured to perform motion compensation on the prediction unit basedon size information of the prediction unit obtained from the headerinformation, wherein the size of the prediction unit is N×N pixels,wherein N is a power of two and N is equal to or more than 32, therebygenerating a prediction block, an inverse-quantizing unit configured toinverse-quantize the received bit stream, an inverse-transforming unitconfigured to inverse-transform the inverse-quantized data to obtain aresidue, and an adder configured to add the reside to the predictionblock to restore the image. The prediction unit has a size of anextended macro-block. The inverse-transforming unit is configured toperform inverse DCT on the extended macro-block. The prediction unit hasa size of N×N pixels, wherein N is a power of two, and wherein the sizeof the prediction unit is restricted to 64×64 pixels or less inconsideration of complexity of an encoder and a decoder. The predictionunit corresponds to a leaf coding unit when a coding unit having avariable size is hierarchically split and reaches a maximum permissiblelevel or level depth. The motion compensating unit is configured tosplit the prediction unit into partitions based on partition informationof the prediction unit and to perform the motion compensation on thesplit partitions. The partition splitting is achieved by an asymmetricpartitioning method. The partition splitting is achieved by ageometrical partitioning method with a shape other than a square shape.The partition splitting is achieved by a partitioning method along adirection of an edge.

To achieve the third object, according to an aspect of the presentinvention, there is provided a method of encoding an image, the methodcomprising the steps of performing intra-prediction encoding byselectively using one of a plurality of prediction modes on a predictionunit split by applying at least one of asymmetric partitioning andgeometrical partitioning to an input image so as to prediction-encodethe input image and performing transform, quantization, and entropyencoding on a residue which is a difference between a prediction unitpredicted by the intra prediction and a current prediction unit. Thepixel value in the asymmetric-partitioned prediction unit may bepredicted from the pixel value in the encoded block earlier than theprediction unit along one prediction direction of vertical direction,horizontal direction, average prediction, diagonal down-right, anddiagonal down-right.

To achieve the fourth object, according to an aspect of the presentinvention, there is provided a method of decoding an image, the methodcomprising the steps of restoring a residue by entropy-decoding areceived bit stream and by performing inverse quantization and inversetransform on the residue, generating a prediction unit by performingintra-prediction encoding that selectively uses one of a plurality ofprediction modes on a prediction unit split by applying at least one ofasymmetric partitioning and geometrical partitioning, and restoring theimage by adding the residue to the prediction.

The pixel value in the asymmetric-partitioned prediction unit may bepredicted from the pixel value in the encoded block earlier than theprediction unit along one prediction direction of vertical direction,horizontal direction, average prediction, diagonal down-right, anddiagonal down-right. The pixel value in the asymmetric-partitionedprediction unit may be predicted from the pixel value in the blockencoded earlier than the prediction unit along the line made at apredetermined even interval in all over the direction of 360°. The pixelvalue in the asymmetric-partitioned prediction unit may be subjected tointra prediction along the line having an angle corresponding to a slopebased on dx and dy which define the slope with dx along the horizontaldirection and dy along the vertical direction. The prediction pixelvalue of the rightmost and lowermost pixel in the prediction unit may beobtained based on the vertically and horizontally corresponding pixelsin the left and upper blocks encoded earlier than the prediction unit.The prediction pixel value of the rightmost and lowermost pixel in theprediction unit may be obtained by linear interpolation and usingvertically and horizontally corresponding inner pixels in the left andupper blocks earlier than the prediction unit. The prediction pixelvalue of the rightmost and lowermost pixel in the current predictionunit of the Nth picture may be obtained through linear interpolationbetween or based on an average value between vertically and horizontallycorresponding pixels in the previously encoded left and upper blocksadjacent to the current unit and vertically and horizontallycorresponding pixels in the previously encoded left and upper blocksadjacent to the prediction unit corresponding to the N−1th picture.

To achieve the fourth object, according to another aspect of the presentinvention, there is provided an apparatus of decoding an imagecomprising an inverse-transforming unit configured to restore a residueby entropy-decoding a received bit stream and by performing inversequantization and inverse transform on the residue, an intra predictingunit configured to generate a prediction unit by performingintra-prediction encoding that selectively uses one of a plurality ofprediction modes on a prediction unit split by applying at least one ofasymmetric partitioning and geometrical partitioning, and an adderconfigured to restore the image by adding the residue to the prediction.

To achieve the fifth object, according to an aspect of the presentinvention, there is provided a method of decoding an image, the methodcomprising the steps of receiving a bit stream in which an intraprediction mode on a current block is encoded, wherein the intraprediction mode is determined based on a residue between a pixeladjacent to the current block having a second size in an Nth picture anda pixel adjacent to a reference block in an N−1th picture temporallypositioned before the Nth picture, obtaining a motion vector, the intraprediction mode and a quantized residue by entropy-decoding the bitstream, obtaining the residue by inverse-quantizing andinverse-transforming the quantized residue, determining a referenceblock of the current block having the second size in at least onepicture by using the motion vector, and restoring the current block byapplying the intra prediction mode to a result of an operation between apixel adjacent to the determined reference block and the residue. Thestep of determining the reference block of the current block having thesecond size in the at least one picture using the motion vector includesdetermining a reference macro-block of the current macro-block havingthe first size including the current block having the second size usingthe motion vector and determining the reference block included in thereference macro-block based on the position of the current blockincluded in the current macro-block. The current macro-block having thefirst size may have a size of 32×32 pixels or more, and the currentblock having the second size may have one of 4×4 and 8×8 pixels.

To achieve the fifth object, according to another aspect of the presentinvention, there is provided a method of decoding an image, the methodcomprising the steps of receiving a bit stream in which an intraprediction mode on a current block is encoded, wherein the intraprediction mode is determined based on a residue between a pixeladjacent to the current block having a second size in an Nth picture anda pixel adjacent to a reference block in an N+1th picture temporallypositioned behind the Nth picture, obtaining a motion vector, the intraprediction mode and a quantized residue by entropy-decoding the bitstream, obtaining the residue by inverse-quantizing andinverse-transforming the quantized residue, determining a referenceblock of the current block having the second size in at least onepicture by using the motion vector, and restoring the current block byapplying the intra prediction mode to a result of an operation between apixel adjacent to the determined reference block and the residue.

To achieve the fifth object, according to another aspect of the presentinvention, there is provided a method of decoding an image, the methodcomprising the steps of receiving a bit stream in which an intraprediction mode on a current block is encoded, wherein the intraprediction mode is determined based on a residue determined based on aforward residue between a pixel adjacent to the current block having asecond size in an Nth picture and a pixel adjacent to a reference blockin an N−1th picture temporally positioned before the Nth picture and areverse residue between a pixel adjacent to the current block having asecond size in an Nth picture and a pixel adjacent to a reference blockin an N+1th picture temporally positioned behind the Nth picture,obtaining a motion vector, the intra prediction mode and a quantizedresidue by entropy-decoding the bit stream, obtaining the residue byinverse-quantizing and inverse-transforming the quantized residue,determining a reference block of the current block having the secondsize in at least one picture by using the motion vector, and restoringthe current block by applying the intra prediction mode to a result ofan operation between a pixel adjacent to the determined reference blockand the residue.

To achieve the fifth object, according to another aspect of the presentinvention, there is provided a method of decoding an image, the methodcomprising the steps of receiving a bit stream in which an intraprediction mode on a current block is encoded, wherein the intraprediction mode is determined based on a residue determined based on afirst residue between a pixel adjacent to the current block having asecond size in an Nth picture and a pixel adjacent to a reference blockin an N−1th picture temporally positioned before the Nth picture and asecond residue between a pixel adjacent to the current block having asecond size in an Nth picture and a pixel adjacent to a reference blockin an N−2th picture temporally positioned before the N−1 th picture,obtaining a motion vector, the intra prediction mode and a quantizedresidue by entropy-decoding the bit stream, obtaining the residue byinverse-quantizing and inverse-transforming the quantized residue,determining a reference block of the current block having the secondsize in at least one picture by using the motion vector, and restoringthe current block by applying the intra prediction mode to a result ofan operation between a pixel adjacent to the determined reference blockand the residue.

To achieve the sixth object, according to an aspect of the presentinvention, there is provided an apparatus of decoding an imagecomprising an entropy decoding unit configured to generate a motionvector, an intra prediction mode, and a quantized residue byentropy-decoding a bit stream in which the intra prediction mode isencoded, wherein the intra prediction mode is determined based on aresidue between a pixel adjacent to the current block having a secondsize in an Nth picture and a pixel adjacent to a reference block in atleast one reference picture, a decoding controller configured to obtaina block size and reference picture information from entropy-decodedinformation, an inverse-quantizing unit configured to inverse-quantizethe quantized residue, an inverse-transforming unit configured toinverse-transform the inverse-quantized reside, and a predicting unitconfigured to determine a reference block of the to-be-decoded currentblock having the second size based on the motion vector and thereference picture information, to operate a pixel adjacent to thedetermined reference block with the residue, and to apply the intraprediction mode to a result of the operation, thereby restoring thecurrent block.

Advantageous Effects

As described above, the method of encoding/decoding high-resolutionimages and apparatus of performing the method set the size of aprediction unit to be encoded to 32×32 pixels, 64×64 pixels, or 128×128pixels, performing motion prediction, motion compensation, andtransformation on the basis of a set prediction unit size. In addition,the method and apparatus of the present invention divide a macro-blockhaving a pixel size of 32×32, 64×64, or 128×128 into at least onepartition on the basis of an edge and perform encoding processes foreach partition.

In the case of a high-homogeneity or high-uniformity region which hasthe same color or has energy concentrated toward the low-frequency, thesize of the prediction unit is expanded to a size of 32×32 pixels, 64×64pixels, or 128×128 pixels, which corresponds to the size of an extendedmacro-block, for application to encoding/decoding, thereby increasingencoding/decoding efficiency of large-size images having an HD or ultraHD or higher resolution.

Further, according to the temporal frequency characteristics (shiftbetween previous and current screen images or degree of motion) forlarge-size images, the size of the extended macro-block is increased ordecreased by using the size of the extended macro-block as the size ofthe prediction unit, thus enhancing encoding/decoding efficiency.

Accordingly, it can be possible to enhance efficiency of encodinglarge-size images having an HD or ultra HD or higher resolution as wellas to reduce encoding noise in a high-homogeneity, high-uniformityregion.

The intra-prediction encoding/decoding method and apparatus can enhanceencoding efficiency for high-resolution images having an HD or ultra HDresolution by applying intra-prediction encoding/decoding to pixelsblocks having an asymmetrical shape or any geometrical shape with a sizeof M×N.

According to the above-described image encoding/decoding method andapparatus performing the same, a residue between a pixel adjacent to thecurrent block having the second size in the Nth picture to be encodedand a pixel adjacent to the reference block having the second sizeincluded in at least one reference picture of the N−2th, N−1th, N+1th,and N+2th reference pictures is obtained, and based on the obtainedresidue, an inter prediction mode is determined, and transform,quantization, and entropy encoding are performed, and the result is thentransferred. Header information such as block size information andreference picture information is also encoded and transferred, therebyenhancing encoding efficiency.

The above-described encoding/decoding method applies toencoding/decoding of each extended macro-block having a size of 32×32pixels, thereby enhancing encoding/decoding efficiency for large-sizeimages having an ultra HD or higher resolution.

DESCRIPTION OF DRAWINGS

FIG. 1 is a flowchart illustrating an image encoding method according toan embodiment of the present invention.

FIG. 2 is a conceptual view illustrating a recursive coding unitstructure according to another embodiment of the present invention.

FIGS. 3 to 6 are conceptual views illustrating an asymmetricpartitioning method according to an embodiment of the present invention.

FIGS. 7 to 9 are conceptual views illustrating a geometricalpartitioning method according to other embodiments of the presentinvention.

FIG. 10 is a conceptual view illustrating motion compensation forboundary pixels located at a boundary line when geometrical partitioningapplies.

FIG. 11 is a flowchart illustrating an image encoding method accordingto another embodiment of the present invention.

FIG. 12 is a conceptual view illustrating a partitioning process shownin FIG. 11.

FIG. 13 is a conceptual view illustrating an example of edge-consideredpartitioning applies to intra prediction.

FIG. 14 is a flowchart illustrating an image encoding method accordingto another embodiment of the present invention.

FIG. 15 is a flowchart illustrating an image encoding method accordingto another embodiment of the present invention.

FIG. 16 is a flowchart illustrating an image decoding method accordingto an embodiment of the present invention.

FIG. 17 is a flowchart illustrating an image decoding method accordingto another embodiment of the present invention.

FIG. 18 is a block diagram illustrating a configuration of an imageencoding apparatus according to an embodiment of the present invention.

FIG. 19 is a block diagram illustrating a configuration of an imageencoding apparatus according to another embodiment of the presentinvention.

FIG. 20 is a block diagram illustrating a configuration of an imagedecoding apparatus according to an embodiment of the present invention.

FIG. 21 is a block diagram illustrating a configuration of an imagedecoding apparatus according to another embodiment of the presentinvention.

FIG. 22 is a conceptual view illustrating an intra-prediction encodingmethod using an asymmetrical pixel block according to an embodiment ofthe present invention.

FIGS. 22 to 25 are conceptual views illustrating an intra-predictionencoding method using an asymmetrical pixel block according to anotherembodiment of the present invention.

FIG. 26 is a conceptual view illustrating an intra-prediction encodingmethod based on planar prediction according to another embodiment of thepresent invention.

FIG. 27 is a conceptual view illustrating an intra-prediction encodingmethod based on planar prediction according to another embodiment of thepresent invention.

FIG. 28 is a block diagram illustrating a configuration of an imageencoding apparatus performing intra-prediction encoding according to anembodiment of the present invention.

FIG. 29 is a flowchart illustrating an image encoding method appliedwith intra-prediction encoding according to an embodiment of the presentinvention.

FIG. 30 is a block diagram illustrating a configuration of an imagedecoding apparatus according to an embodiment of the present invention.

FIG. 31 is a flowchart illustrating an image decoding method accordingto an embodiment of the present invention.

FIG. 32 is a flowchart illustrating an image encoding method accordingto an embodiment of the present invention.

FIG. 33 is a conceptual view illustrating the image encoding methodshown in FIG. 32.

FIG. 34 is a flowchart illustrating an image encoding method accordingto another embodiment of the present invention.

FIG. 35 is a conceptual view illustrating the image encoding methodshown in FIG. 34.

FIG. 36 is a flowchart illustrating an image encoding method accordingto another embodiment of the present invention.

FIG. 37 is a conceptual view illustrating the image encoding methodshown in FIG. 36.

FIG. 38 is a flowchart illustrating an image encoding method accordingto another embodiment of the present invention.

FIG. 39 is a conceptual view illustrating the image encoding methodshown in FIG. 38.

FIG. 40 is a flowchart illustrating an image decoding method accordingto an embodiment of the present invention.

FIG. 41 is a block diagram illustrating a configuration of an imageencoding apparatus according to an embodiment of the present invention.

FIG. 42 is a block diagram illustrating a configuration of an imagedecoding method according to an embodiment of the present invention.

DESCRIPTION OF KEY ELEMENTS IN DRAWINGS

-   -   1810, 1910: prediction unit determining unit 1820, 1920, 2110:        prediction unit partitioning unit    -   1830: encoder    -   1930: decoder

BEST MODE

Various modifications and variations may be made to the presentinvention. Hereinafter, some particular embodiments will be described indetail with reference to the accompanying drawings.

However, it should be understood that the present invention is notlimited to the embodiments and all the variations or replacements of theinvention or their equivalents are included in the technical spirit andscope of the present invention.

It will be understood that, although the terms first, second, third etc.may be used herein to describe various elements, components, regions,layers and/or sections, these elements, components, regions, layersand/or sections should not be limited by these terms. These terms areonly used to distinguish one element, component, region, layer orsection from another region, layer or section. Thus, a first element,component, region, layer or section discussed below could be termed asecond element, component, region, layer or section without departingfrom the teachings of the present invention

It will be understood that when an element or layer is referred to asbeing “on,” “connected to” or “coupled to” another element or layer, itcan be directly on, connected or coupled to the other element or layeror intervening elements or layers may be present. In contrast, when anelement is referred to as being “directly on,” “directly connected to”or “directly coupled to” another element or layer, there are nointervening elements or layers present. Like numerals refer to likeelements throughout. As used herein, the term “and/or” includes any andall combinations of one or more of the associated listed items.

The terminology used herein is for the purpose of describing particularexample embodiments only and is not intended to be limiting of thepresent invention. As used herein, the singular forms “a,” “an” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. It will be further understood thatthe terms “comprises” and/or “comprising,” when used in thisspecification, specify the presence of stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

Hereinafter, embodiments of the present invention will be described indetail with reference to the accompanying drawings, wherein the samereference numerals may be used to denote the same or substantially thesame elements throughout the specification and the drawings, anddescription on the same elements will be not repeated.

FIG. 1 is a flowchart illustrating an image encoding method according toan embodiment of the present invention. FIG. 1 illustrates a method ofdetermining the size of a macro-block according to temporal frequencycharacteristics of an image and of then performing motion compensationencoding by using the determined size of macro-block.

Referring to FIG. 1, the encoding apparatus receives a target frame (orpicture) to be encoded (step S110). The received target frame may bestored in a buffer which may store a predetermined number of frames. Forexample, the buffer may store at least four (n−3th, n−2th, n−1th, andnth) frames.

Subsequently, the encoding apparatus analyzes temporal frequencycharacteristics of the received frame (or picture) (step S120). Forexample, the encoding apparatus detects a variance between the n−3thframe (or picture) and the n−2th frame (or picture), a variance betweenthe n−2th frame (or picture) and the n−1th frame (or picture), and avariance between the n−1th frame (or picture) and the nth frame (orpicture) to analyze the temporal frequency characteristics betweenframes (or pictures).

Thereafter, the encoding apparatus compares the analyzed temporalfrequency characteristics with a preset threshold value, and based on aresult of the comparison, determines the size of a macro-block to beencoded (step S130). Here, the encoding apparatus may determine the sizeof the macro-block based on a variance between two temporally adjacentframes (for example, n−1th and nth frames) among the frames stored inthe buffer or may determine the size of the macro-block based oncharacteristics of changes in a predetermined number of frames (forexample, n−3th, n2-th, n−1th, and nth frames) to reduce overhead forinformation on the size of the macro-block.

For example, the encoding apparatus analyzes the temporal frequencycharacteristics of the n−1th frame (or picture) and the nth frame (orpicture), and if the analyzed temporal frequency characteristics valueis less than a preset first threshold value, determines the size of themacro-block as 64×64 pixels. If the analyzed temporal frequencycharacteristics value is equal to and more than the preset firstthreshold value and less than a second threshold value, the size of themacro-block is determined as 32×32 pixels, and if the analyzed temporalfrequency characteristics value is equal to or more than the presetsecond threshold value, the size of the macro-block is determined as16×16 pixels or less. Here, the first threshold value refers to atemporal frequency characteristics value when a variance between frames(or pictures) is smaller than the second threshold value. Hereinafter,the extended macro-block is defined as a macro-block having a size of32×32 pixels or more. The extended macro-block may have a size of 32×32pixels or more, such as 64×64 pixels or 128×128 pixels, to be suited forhigh resolution including ultra HD or higher resolution. In case ofultra HD or higher resolution, the extended macro-block may berestricted to the maximum size of 64×64 pixels or less in considerationof complexity of the encoder and decoder.

The size of the macro-block to be encoded may have a predetermined valuefor each picture or each GOP (Group of Pictures) based on the result ofanalysis of the temporal frequency characteristics of the receivedpicture.

Or, irrespective of the result of analysis of the temporal frequencycharacteristics of the received picture, the size of the macro-block tobe encoded may have a predetermined value for each picture or each GOP.

If the size of the macro-block is determined in step S130, the encodingapparatus performs encoding on the basis of the determined size ofmacro-block (step S140).

For example, if the size of the macro-block is determined to be 64×64pixels, the encoding apparatus performs motion prediction on a currentmacro-block whose size is 64×64 pixels to obtain a motion vector whichis then used to conduct motion compensation to generate a predictionblock, and transforms, quantizes and performs entropy encoding on aresidue, which is a difference between the generated prediction blockand the current macro-block, then transmits the result. Further,information on the determined size of the macro-block and information onthe motion vector are also encoded and then transmitted.

In the following embodiments, per-extended macro-block encoding may befulfilled depending on the size of the macro-block which is determinedby an encoding controller (not shown) or a decoding controller (notshown), and as described above, may apply to all of the motioncompensation encoding, transform, and quantization. However, it mayapply to at least one of the motion compensation encoding, transform, orquantization. Further, the per-extended macro-block encoding may applylikewise to decoding according to the following embodiments.

As shown in FIG. 1, in the image encoding method according to anembodiment, when a variance between input frames is small (i.e., whenthe temporal frequency is low), the size of the macro-block increasesand when the variance between frames is large (i.e., when the temporalfrequency is high), the size of the macro-block decreases, therebyenhancing encoding efficiency.

The above-described image encoding/decoding method according to temporalfrequency characteristics may apply high-resolution images having HD orultra HD or higher resolution. Hereinafter, the “macro-block” refers toan extended macro-block or an existing macro-block that has a size of32×32 pixels or less.

According to another embodiment of the present invention, instead ofperforming encoding/decoding based on the extended macro-block or itssize, a recursive coding unit (CU) may be used to perform encoding anddecoding. Hereinafter, a recursive coding unit structure according to anembodiment is described with reference to FIG. 2.

FIG. 2 is a conceptual view illustrating a recursive coding unitstructure according to an embodiment of the present invention.

Referring to FIG. 2, each coding unit CU has a square shape and may havea variable size of 2N×2N (unit: pixels). Inter prediction, intraprediction, transform, quantization, and entropy encoding may beperformed on a per-coding unit basis. The coding unit CU may include amaximum coding unit LCU and a minimum coding unit SCU. The size of themaximum or minimum coding unit LCU or SCU may be represented by powersof 2 which are 8 or more.

According to an embodiment, the coding unit CU may have a recursive treestructure. FIG. 2 illustrates an example where a side of the maximumcoding unit LCU (or CU₀) has a size of 2N₀ which is 128 (N₀=64) whilethe maximum level or level depth is 5. The recursive structure may berepresented by a series of flags. For example, in the case that a codingunit CU_(k) whose level or level depth is k has a flag value of 0,coding on the coding unit CU_(k) is performed on the current level orlevel depth. When the flag value is 1, the coding unit CU_(k) is splitinto four independent coding units CU_(k+1) having a level or leveldepth of k+1 and a size of N_(k+1)×N_(k+1). In this case, the codingunit CU_(k+1) may be recursively processed until its level or leveldepth reaches the permissible maximum level or level depth. When thelevel or level depth of the coding unit CU_(k+1) is the same as thepermissible maximum level or level depth (which is, e.g., 4 as shown inFIG. 4), any further splitting is not permissible.

The size of the maximum coding unit LCU and the size of the minimumcoding unit SCU may be included in a sequence parameter set (SPS). Thesequence parameter set SPS may include the permissible maximum level orlevel depth of the maximum coding unit LCU. For example, in the exampleillustrated in FIG. 2, the permissible maximum level or level depth is5, and when the side of the maximum coding unit LCU has a size of 128pixels, five coding unit sizes, such as 128×128 (LCU), 64×64, 32×32,16×16, and 8×8 (SCU), may be possible. That is, given the size of themaximum coding unit LCU and the permissible maximum level or leveldepth, the permissible size of the coding unit may be determined.

The size of the coding unit, in the case of high resolution, such asultra HD, may be restricted to the maximum size of 64×64 inconsideration of complexity of the encoder and decoder.

Use of the above-described recursive coding unit structure may providethe following advantages.

First, a size larger than that of the existing 16×16 macro-block may besupported. If an image region of interest is homogeneous, the maximumcoding unit LCU may express the image region of interest with a smallernumber of symbols than when using a number of small blocks.

Second, compared to when using a fixed size of macro-block, any size ofmaximum coding unit LCU may be supported, so that the codec may beeasily optimized to various contents, applications, and apparatuses.That is, the size of the maximum coding unit LCU, the maximum level orlevel depth may be properly selected so that the hierarchical blockstructure may be optimized further than the target application.

Third, irrespective of whether it is a macro-block, sub-macro-block, orextended macro-block, a single unit type of a coding unit LCU is used sothat the multilevel hierarchical structure may be simply represented byusing the size of the maximum coding unit LCU, the maximum level (or themaximum level depth), and a series of flags. When used together withsize-independent syntax representation, the coding unit LCU is enough toindicate one generalized size of syntax item for the remaining codingtools, and such conformity may simplify actual parsing processes. Themaximum level value (or maximum level depth value) may be any value andmay have a value larger than a value permitted in the existing H.264/AVCencoding scheme. All syntax elements may be indicated in a consistentmanner independent from the size of the coding unit CU by using thesize-independent syntax representation. The splitting process for thecoding unit may be recursively indicated, and syntax elements for theleaf coding unit (the last coding unit in the level) may be defined tothe same size regardless of the size of the coding unit. The aboverepresentation is very effective in reducing parsing complexity and maymake the representation further clarified when a high level or leveldepth is allowed.

If the hierarchical splitting process is complete, inter prediction orintra prediction may be performed on the leaf node of the coding unithierarchical unit without being further split. This leaf coding unit isused as the prediction unit PU which is a basic unit of the interprediction or intra prediction.

For inter prediction or intra prediction, partitioning is fulfilled onthe leaf coding unit. That is, partitioning is performed on theprediction unit PU. Here, the prediction unit PU is a basic unit forinter prediction or intra prediction and may be an existing macro-blockunit or sub-macro-block unit, or an extended macro-block unit having asize of 32×32 pixels or more or a coding unit.

Partitioning includes asymmetrical partitioning, geometricalpartitioning in any shape other than square, and partitioning along anedge direction, which are now described in greater detail.

FIGS. 3 to 6 are conceptual views illustrating asymmetric partitioningaccording to an embodiment.

When the prediction unit PU for inter prediction or intra prediction hasa size of M×M (M is a natural number. The unit of the size is pixel),asymmetric partitioning is performed along a horizontal or verticaldirection of the coding unit. FIG. 3 illustrates an example where thesize of the prediction unit PU is 64×64 pixels.

Referring to FIG. 3, asymmetric partitioning is conducted along ahorizontal direction to split the prediction unit into a partition P11 ahaving a size of 64×16 and a partition P21 a having a size of 64×48 orinto a partition P12 a having a size of 64×48 and a partition P22 ahaving a size of 64×16. Or, asymmetric partitioning is performed along avertical direction to split the prediction unit into a partition P13 ahaving a size of 16×64 and a partition P23 a having 48×64 or into apartition P14 a having a size of 48×64 and a partition P24 a having asize of 16×64.

Referring to FIG. 4, in the case of having a size of 32×32, theprediction unit may be subjected to horizontal-direction asymmetricpartitioning to be split into a partition P11 b having a size of 32×8and a partition P21 b having a size of 32×24 or into a partition P12 bhaving a size of 32×24 and a partition P22 b having a size of 32×8. Or,the prediction unit may be subjected to vertical-direction asymmetricpartitioning to be split into a partition P13 b having a size of 8×32and a partition P23 b having a size of 24×32 or into a partition P14 bhaving a size of 24×32 and a partition P24 b having a size of 8×32.

Referring to FIG. 5, in the case of having a size of 16×16, theprediction unit PU may be subjected to horizontal-direction asymmetricpartitioning to be split into a partition P11 c having a size of 16×4and a partition P21 c having a size of 16×12 or (although not shown inthe drawings) into an upper partition having a size 16×12 and a lowerpartition having a size of 16×4. Further, although not shown in thedrawings, the prediction unit PU may be subjected to vertical-directionasymmetric partitioning to be split into a left partition having a sizeof 4×16 and a right partition having a size of 12×16 or into a leftpartition having a size of 12×16 and a right partition having a size of4×16.

Referring to FIG. 6, in the case of having a size of 8×8, the predictionunit PU may be subjected to horizontal-direction asymmetric partitioningto be split into a partition P11 d having a size of 8×2 and a partitionP21 d having a size of 8×6 or (although not shown in the drawings) intoan upper partition having a size 8×6 and a lower partition having a sizeof 8×2. Further, although not shown in the drawings, the prediction unitPU may be subjected to vertical-direction asymmetric partitioning to besplit into a left partition having a size of 2×8 and a right partitionhaving a size of 6×8 or into a left partition having a size of 6×8 and aright partition having a size of 2×8.

FIGS. 7 to 9 are conceptual views illustrating geometrical partitioningaccording to another embodiment of the present invention.

FIG. 7 illustrates an example where the prediction unit PU is subjectedto geometrical partitioning so that the split partitions have a shapeother than square.

Referring to FIG. 7, for the prediction unit, a geometrical boundaryline L between partitions may be defined as follows. The prediction unitPU is divided into four quadrants by x and y axes passing through thecenter O of the prediction unit PU. A vertical line is drawn from thecenter O to the boundary line L. Then, any possible boundary line Lpositioned in any direction may be specified by a distance ρ from thecenter O to the boundary line L and an angle θ from the x axis to thevertical line in a counterclockwise direction.

FIG. 8 illustrates an example where the prediction unit PU is subjectedto geometrical partitioning so that each split partition has any shapeother than square.

Referring to FIG. 8, for inter or intra prediction, the prediction unitPU is divided into four quadrants with respect to its center. The secondquadrant which is the upper and left portion of the prediction unit PUis split into a partition P11 b′, and the remaining L-shaped quadrantsare split into a partition P21 b′. As used herein, the “portion” of theprediction unit PU, which corresponds to a split partition or severalsplit partitions, is also called “block”. Or, the third quadrant whichis the lower and left portion of the prediction unit PU is split into apartition P12 b′, and the remaining quadrants are split into a partitionP22 b′. Alternatively, the first quadrant which is the upper and rightportion of the prediction unit PU is split into a partition P13 b′, andthe remaining quadrants are split into a partition P23 b′. Also, thelower and right portion of the prediction unit PU which corresponds tothe fourth quadrant is split into a partition P14 b′, with the remainingquadrants slit into a partition P23 b′. Further, the fourth quadrant,the lower and right portion of the prediction unit PU, is split into apartition P14 b′, with the remaining quadrants split into a partitionP24 b′.

As described above, the prediction unit may be split so that a splitpartition has an L shape. Accordingly, in the case that, uponpartitioning, there is a moving object in an edge block, e.g., the upperand left, upper and right, lower and right, or lower and left block, itmay provide more effective encoding than when the prediction unit PU issplit into four blocks. Depending on the edge block in which the movingobject is positioned among the four partitions, the correspondingpartition may be selected and used.

FIG. 9 illustrates another example where the prediction unit PU issubjected to geometrical partitioning so that at least one of splitpartitions has a non-square shape.

Referring to FIG. 9, for inter or intra prediction, the prediction unitPU may be split into two different irregular regions (modes 0 and 1) orinto differently sized rectangular regions (modes 2 and 3).

Here, the parameter “pos” is used to indicate a position of a boundarybetween partitions (shortly “inter-partition boundary). In the case ofmode 0 or 1, “pos” denotes a horizontal-direction distance from adiagonal line of the prediction unit PU to the inter-partition boundary,and in the case of mode 2 or 3, “pos” denotes a horizontal-directiondistance from a vertical or horizontal bisector to the inter-partitionboundary. In FIG. 9, mode information may be transmitted to the decoder.Among the above-mentioned four modes, in terms of RD (Rate Distortion),a mode having the minimum RD cost may be used for inter prediction.

FIG. 10 is a conceptual view illustrating motion compensation forboundary pixels positioned along a boundary line in the case ofgeometrical partitioning. When the prediction unit PU is split intoregions 1 and 2 by geometrical partitioning, a motion vector for theregion 1 is referred to as MV1, a motion vector for the region 2 as MV2.

When among the specific top, bottom, left, and right pixels positionedin the region 1 (or 2), any one is also positioned in the region 2 (or1)), it can be deemed a boundary pixel. Referring to FIG. 10, a boundarypixel A belongs to a boundary between the two regions in the region 2,and a boundary pixel B belongs to a boundary between the two regions inthe region 1. Pixels other than the boundary pixels are subjected tonormal motion compensation using an appropriate motion vector. Theboundary pixels undergo motion compensation using a sum of valuesobtained by multiplying weighted factors by motion prediction valuesfrom the motion vectors MV1 and MV2 of the regions 1 and 2,respectively. In the example illustrated in FIG. 10, a weighted factorof ⅔ is used for a region including the boundary pixels, and a weightedfactor of ⅓ is used for a region including no boundary pixels.

FIG. 11 is a flowchart illustrating an image coding method according toanother embodiment of the present invention. FIG. 12 is a conceptualview illustrating the partitioning illustrated in FIG. 11.

FIG. 11 illustrates a process in which after the size of the predictionunit PU is determined by the image encoding method shown in FIG. 1, theprediction unit PU is split into partitions considering the edgeincluded in the prediction unit PU, and encoding is then performed foreach split partition. An example is described in which the predictionunit PU uses a macro-block whose size is 32×32.

The edge-considered partitioning may apply to intra prediction as wellas the inter prediction, and its detailed description will be givenbelow in detail.

Steps S1110 to 1130 in FIG. 11 are substantially the same as steps S110to S130, respectively, in FIG. 1, and thus detailed description is notrepeated.

Referring to FIGS. 11 and 12, the size of the macro-block is determinedin steps S1110 to S1130. Then, the encoding apparatus detects pixelsbelonging to an edge among pixels included in a macro-block adjacent toa current macro-block having the determined size (step S1140).

For detecting the pixels belonging to the edge in step S1140, variousknown methods may be employed. For example, difference values betweenthe current macro-block and its adjacent pixels may be calculated or“sobel” algorithm or other edge-detection algorithms may be used foredge detection.

Thereafter, the encoding apparatus uses the pixels belonging to thedetected edge to split the current macro-block into partitions (stepS1150).

For splitting of the current macro-block, the encoding apparatus detectsa pixel belonging to the edge among pixels adjacent to an edge pixel,which is determined to belong to the edge among pixels included inblocks adjacent to the current macro-block, and uses a line connectingthe detected pixel with the edge pixel.

For example, as shown in FIG. 12, the encoding apparatus detects pixels211 and 214 belonging to the edge from the closet pixels among pixelsbelonging to the blocks adjacent the current macro-block having a sizeof 32×32 pixels. Then, the encoding apparatus detects a pixel 212belonging to the edge among pixels positioned around the detected pixel211 and uses an extension line 213 connecting the pixels 211 and 212,thereby splitting the macro-block into partitions.

Or, the encoding apparatus detects a pixel 215 belonging to the edgeamong pixels adjacent to the detected pixel 214 and uses an extensionline connecting the pixels 214 and 215 with each other, therebypartitioning the macro-block.

Further, the encoding apparatus detects, among only the pixels closestto the current macro-block 210 as the ones included in blocks adjacentto the current macro-block 210, pixels belonging to the edge (alsoreferred to as “edge pixel(s)”), and determines the direction of astraight line passing through the edge pixels (this straight line isalso referred to as “edge straight line”), thereby partitioning thecurrent macro-block. Here, with respect to the edge straight line, amongintra prediction modes for 4×4 blocks according to H.264/AVC standards,such as vertical mode (mode 0), horizontal mode (mode 1), diagonaldown-left mode (mode 3), diagonal down-right mode (mode 4), verticalright mode (mode 5), horizontal-down mode (mode 6), vertical left mode(mode 7), and horizontal-up mode (mode 8), any one may be used topartition the current macro-block. Alternatively, partitions split indifferent directions with respect to the edge pixels are encoded, andconsidering encoding efficiency, the final direction of the straightline may be then determined. Or, in relation to the direction of theedge straight line, rather than the intra prediction modes for 4×4blocks based on H.264/AVC standards, various intra predictions modes forblocks having a size larger than 4×4 pixel size are used among which anyone is chosen, and the current macro-block may be subjected topartitioning along the direction indicated by the chosen mode.Information on the edge straight line (e.g., directional information)may be included in the partition information and may be transferredtogether with the partition information.

By the above-described method, the current macro-block is split into atleast one partition in step S150. Then, the encoding apparatus performsencoding for each partition (step S1160).

For instance, the encoding apparatus performs motion prediction on eachsplit partition in the current macro-block having a size of 64×64 or32×32 pixels to obtain a motion vector and performs motion compensationusing the obtained motion vector to generate a prediction partition.Then the encoding apparatus conducts transform, quantization, andentropy encoding on a residue which is a difference between thegenerated prediction partition and the partition of the currentmacro-block, then transfers the result. The encoding apparatus alsoperforms entropy encoding on information regarding the size, partitioninformation, and motion vector of the determined macro-block and thentransfers the result.

The inter prediction using the above-described edge-consideredpartitioning may be made to be carried out when the prediction modeusing the edge-considered partitioning is activated. As described abovethe edge-considered partitioning may apply to intra prediction as wellas the inter prediction. An example of applying to the intra predictionis described with reference to FIG. 13.

FIG. 13 is a conceptual view illustrating an example of applyingedge-considered partitioning to intra prediction. As shown in FIG. 13,intra prediction using the edge-considered partitioning may beconfigured to be performed when the prediction mode using theedge-considered partitioning is activated. An edge is detected using theabove-mentioned edge detection algorithm, such as sobel algorithm, andreference pixels may be then estimated along the detected edgedirection.

Referring to FIG. 13, when a line E is an edge boundary line, pixels aand b are pixels positioned at both sides of the edge boundary line E,and p(x, y) is a reference pixel targeted for intra prediction, p(x, y)may be predicted according to Equation 1:Wa=δx−floor(δx)Wb=ceil(δx)−δxP=WaXa+WbXb  [Equation 1]

Here, δx refers to a distance from the x-axis coordinate of referencepixel p(x, y) to the intersection between the edge line E and the xaxis, Wa and Wb to weighted factors, floor(δx) returning a largestinteger not more than δx—for example, floor(1.7)=1—and ceil(δx)returning a value obtained by rounding off δx—for example, ceil(1.7)=2.

Information (including directional information) on the edge boundaryline passing through the pixels included in the edge is included in thepartition information and transferred to the decoder.

FIG. 14 is a flowchart illustrating an image encoding method accordingto another embodiment of the present invention. FIG. 14 illustrates amethod of performing motion compensation encoding using a predictionunit PU having a size determined depending on spatial frequencycharacteristics of an image.

Referring to FIG. 14, the encoding apparatus first receives a targetframe (or picture) to be encoded (step S1310). Here, the received targetframe may be stored in the buffer which may store a predetermined numberof frames. For example, the buffer may store at least four (n−3th,n−2th, n−1th, and nth) frames.

Then, the encoding apparatus analyzes spatial frequency characteristicsof each received frame (or picture) (step S1420). For example, theencoding apparatus calculates signal energy of each frame stored in thebuffer, analyzes a relationship between the calculated signal energy andthe frequency spectrum, and analyze spatial frequency characteristics ofeach image.

Thereafter, the encoding apparatus determines the size of the predictionunit PU based on the analyzed spatial frequency characteristics. Here,the size of the prediction unit PU may be determined on the basis ofeach frame stored in the buffer or on the basis of a predeterminednumber of frames.

For example, the encoding apparatus determines the size of theprediction unit PU to 16×16 pixels or less when the signal energy of theframe (or picture) is less than the third threshold value preset in thefrequency spectrum, to 32×32 pixels when the signal energy is equal toor more than the preset third threshold value and less than the fourththreshold value, and to 64×64 pixels when the signal energy is equal toor more than the preset fourth threshold value. Here, the thirdthreshold value refers to when the spatial frequency of the image ishigher than that in case of the fourth threshold value.

According to the temporal or spatial frequency characteristics of eachreceived picture, the size of the macro-block is used for encoding bythe extended macro-block or prediction unit, so that encoding efficiencymay be enhanced. However, according to the resolution (size) of eachpicture received independently from the temporal or spatial frequencycharacteristics of each received picture, encoding/decoding may be doneusing the extended macro-block or prediction unit. In other words,encoding/decoding may be conducted on pictures having a HD or ultra HDor higher resolution by using the extended macro-block or prediction.

If the size of the prediction unit PU is determined in step S1330, theencoding apparatus performs encoding on the basis of the determined sizeof prediction unit PU (step S1440).

For instance, if the size of the prediction unit PU is determined to64×64 pixels, the encoding apparatus performs motion prediction on thecurrent prediction unit PU having a size of 64×64 pixels to obtain amotion vector and performs motion compensation based on the obtainedmotion vector to generate a prediction block. Then, the encodingapparatus conducts transform, quantization, and entropy encoding on aresidue which is a difference between the generated prediction block andthe current prediction unit PU, and then transfers the result. Further,information on the size of the determined prediction unit PU andinformation on the motion vector are also subjected to entropy encodingand then transferred.

As shown in FIG. 14, in the image encoding method according to anembodiment, when image homogeneity or uniformity of an input picture ishigh (that is, when spatial frequency is low—for example, such as aregion having the same color or a region whose energy is concentratedtoward the lower frequency), the size of the prediction unit PU is setto be larger, e.g., to 32×32 pixels or more, and when the imagehomogeneity or uniformity of a picture is low (that is, when spatialfrequency is high), the size of the prediction unit PU is set to belower, e.g., to 16×16 pixels or less, thereby enhancing encodingefficiency.

FIG. 15 is a flowchart illustrating an image encoding method accordingto another embodiment. FIG. 15 illustrates a process in which after thesize of the prediction unit PU is determined according to the imageencoding method illustrated in FIG. 14, considering an edge included inthe prediction unit PU having the determined size, the prediction unitPU is split to partitions, and for each partition, encoding isfulfilled.

Steps S1510 to S1530 in FIG. 15 are substantially the same as stepsS1410 to S1430 in FIG. 14, and thus their description is not repeated.

Referring to FIG. 15, if the size of the prediction unit PU according tothe spatial frequency characteristics is determined in steps S1510 toS1530, the encoding apparatus detects pixels belonging to the edge amongpixels included in a prediction unit PU adjacent to the currentprediction unit having the determined size (step S1540).

The detection of the pixels belonging to the edge in step S1540 may beperformed by various known methods. For example, a difference valuebetween the current prediction unit PU and its adjacent pixels may becalculated or an edge may be detected based on an edge detectionalgorithm, such as, e.g, sobel algorithm.

Thereafter, the encoding apparatus uses the pixels belonging to thedetected edge to split the current prediction unit PU into partitions(step S1550).

As shown in FIG. 3, the encoding apparatus, to split the currentprediction unit PU, may detect pixels belonging to the edge among pixelsadjacent to edge pixels detected from pixels included in a surroundingblock adjacent to the current prediction unit PU and may then dopartitioning by using a line connecting the pixels adjacent to thedetected edge with the edge pixels detected in step S1540.

Or, the encoding apparatus may split the current prediction unit PU bydetecting pixels belonging to the edge among only pixels closest to thecurrent prediction unit PU among pixels included in the block adjacentto the current prediction unit PU and then determining the direction ofa line passing through the pixels included in the detected edge.

If the current prediction unit PU is split into at least one partitionin step S1550 by the above-described method, the encoding apparatusperforms encoding for each partition (step S360).

For example, the encoding apparatus performs motion prediction on eachpartition split in the current prediction unit PU having a size of 64×64or 32×32 pixels to obtain a motion vector and performs motioncompensation using the obtained motion vector to generate a predictionpartition. Then the encoding apparatus conducts transform, quantization,and entropy encoding on a residue which is a difference between thegenerated prediction partition and the partition of the currentprediction unit PU, and then transfers the result. Further, informationon the size of the determined prediction unit PU, partition information,and information on the motion vector are subjected to entropy encodingand then transferred.

The edge-considered partitioning described in connection with FIG. 15may apply to intra prediction described in connection with FIG. 13 aswell as inter prediction.

FIG. 16 is a flowchart illustrating an image decoding method accordingto an embodiment of the present invention.

Referring to FIG. 16, the decoding apparatus first receives a bit streamfrom the encoding apparatus (step S1610).

Thereafter, the decoding apparatus performs entropy decoding on thereceived bit stream to obtain information on the current prediction unitPU to be decoded (step S1620). Here, instead of performing encoding anddecoding using the extended macro-block and the size of the extendedmacro-block, the above-described recursive coding unit CU may be used todo encoding and decoding, in which case, the prediction unit PUinformation may include the size of the maximum coding unit LCU, thesize of the minimum coding unit SCU, the maximally permissible level orlevel depth, and flag information. Simultaneously, the decodingapparatus obtains a motion vector for motion compensation. Here, thesize of the prediction unit PU may be determined according to thetemporal or spatial frequency characteristics in the encoding apparatusas illustrated in FIGS. 1 and 14—for example, the size may have 32×32 or64×64 pixels. A decoding controller (not shown) receives from theencoding apparatus information on the size of the prediction unit PUapplied by the encoding apparatus and conducts motion compensationdecoding or inverse conversion or inverse quantization based on the sizeof the prediction unit applied by the encoding apparatus.

The decoding apparatus uses information on the size of the predictionunit PU (e.g., 32×32 or 64×64 pixels) and motion vector information aswell as the previously restored picture to generate the predictedprediction unit PU for motion compensation (step S1630).

Then, the decoding apparatus adds the generated predicted predictionunit PU to the residue provided from the encoding apparatus to restorethe current prediction unit PU (step S1640). Here, the decodingapparatus conducts entropy decoding, inverse quantization, and inversetransform on the bit stream provided from the encoding apparatus toobtain the residue. The inverse transform process may be also done onthe basis of the prediction unit (PU) size (e.g., 32×32 or 64×64 pixels)obtained in step S1620.

FIG. 17 is a flowchart illustrating an image decoding method accordingto another embodiment of the present invention. FIG. 17 illustrates aprocess in which the macro-block, which has a size determined accordingto the temporal or spatial frequency characteristics in the imageencoding apparatus, is split along the edge thereby decoding the encodedimage for each split partition.

Referring to FIG. 17, the decoding apparatus receives a bit stream fromthe encoding apparatus (step S1710).

Thereafter, the decoding apparatus performs entropy decoding on thereceived bit stream to obtain the current prediction unit PU informationto be decoded and the partition information of the current predictionunit PU (step S1720). The size of the current prediction unit PU may be32×32 or 64×64 pixels. Simultaneously, the decoding apparatus obtains amotion vector for motion compensation. In the case that, instead ofusing the extended macro-block and the size of the extended macro-blockto perform encoding and decoding, the recursive coding unit CU is usedto do encoding and decoding, the prediction unit PU information mayinclude the size of the maximum coding unit LCU, the size of the minimumcoding unit SCU, the maximum permissible level or level depth, and flaginformation. The partition information may include partition informationtransmitted to the decoder in the case of asymmetric partitioning,geometrical partitioning, and partitioning along the direction of theedge (shortly, referred to as “edge-directional partitioning”).

Next, the decoding apparatus uses the obtained prediction unit PUinformation and the partition information to partition the predictionunit PU (step S1730).

Further, the decoding apparatus generates a prediction partition usingthe partition information, the motion vector information, and thepreviously restored picture (step S1740) and adds the generatedprediction partition to the residue provided from the encodingapparatus, thereby restoring the current partition (step S1750). Here,the decoding apparatus may obtain the residue by performing entropydecoding, inverse quantization, and inverse transform on the bit streamprovided from the encoding apparatus.

Thereafter, the decoding apparatus restores all the partitions includedin the block based on the obtained partition information andreconfigures the restored partitions, then restoring the currentmacro-block (step S1760).

FIG. 18 is a block diagram illustrating a configuration of an imageencoding apparatus according to an embodiment of the present invention.

Referring to FIG. 18, the image encoding apparatus may include aprediction unit determining unit 1810 and an encoder 1830. The encoder1830 may include a motion predicting unit 1831, a motion compensatingunit 1833, an intra predicting unit 1835, a subtractor 1837, atransforming unit 1839, a quantizing unit 1841, an entropy encoding unit1843, an inverse quantizing unit 1845, an inverse transforming unit1847, an adder 1849, and a frame buffer 1851. The prediction unitdetermining unit 1810 may be performed in an encoding controller (notshown) that determines the size of the prediction unit applying to interand intra prediction or may be performed in a separate block outside theencoder. Hereinafter, an example is described in which the predictionunit determining unit 1810 is performed in a separate block outside theencoder.

The prediction unit determining unit 1810 receives a provided inputimage and stores the image in a buffer (not shown) included in theprediction unit determining unit 1810, then analyzing the temporalfrequency characteristics of the stored frame. The buffer may store apredetermined number of frames. For example, the buffer may store atleast four (n−3th, n−2th, n−1th, and nth) frames.

The prediction unit determining unit 1810 may detect a variance betweenthe n−3th frame (or picture) and the n−2th frame (or picture), avariance between the n−2th frame (or picture) and the n−1th frame (orpicture), and a variance between the n−1th frame (or picture) and thenth frame (or picture) to analyze the temporal frequency characteristicsbetween frames (or pictures) and may compare the analyzed temporalfrequency characteristics with a preset threshold value, and based on aresult of the comparison, may determine the size of the prediction unitto be encoded.

Here, the prediction unit determining unit 1810 may determine the sizeof the prediction unit based on a variance between two temporallyadjacent frames (for example, n−1th and nth frames) among the framesstored in the buffer or may determine the size of the prediction unitbased on characteristics of changes in a predetermined number of frames(for example, n−3th, n2-th, n−1th, and nth frames) to reduce overheadfor information on the size of the prediction unit.

For example, the prediction unit determining unit 1810 analyzes thetemporal frequency characteristics of the n−1th frame (or picture) andthe nth frame (or picture), and if the analyzed temporal frequencycharacteristic value is less than a preset first threshold value,determines the size of the prediction unit as 64×64 pixels. If theanalyzed temporal frequency characteristics value is equal to and morethan the preset first threshold value and less than a second thresholdvalue, the size of the prediction unit is determined as 32×32 pixels,and if the analyzed temporal frequency characteristics value is equal toor more than the preset second threshold value, the size of theprediction unit is determined as 16×16 pixels or less. Here, the firstthreshold value refers to a temporal frequency characteristics valuewhen a variance between frames (or pictures) is smaller than the secondthreshold value.

The prediction unit determining unit 1810 provides the prediction unitinformation determined for inter or intra prediction to the entropyencoding unit 1843 and provides it to the encoder 1830 on the basis ofthe prediction unit having a determined size. Here, the prediction unitinformation may include information on the size of the prediction unitdetermined for inter or intra prediction. Specifically, in the case thatencoding and decoding are performed using the extended macro-block andthe size of the extended macro-block, the prediction block informationmay include information on the size of the macro-block or the extendedmacro-block. In the case that the above-mentioned recursive coding unitCU is used to perform encoding and decoding, the prediction unitinformation may include, instead of the information on the size of themacro-block, information on the size of the maximum coding unit LCU tobe used for inter or intra prediction, that is, the size of theprediction unit, and further, the prediction unit information mayinclude the size of the maximum coding unit LCU, the size of the minimumcoding unit SCU, the maximum permissible level or level depth, and flaginformation.

The prediction unit determining unit 1810 may determine the size of theprediction unit by analyzing the temporal frequency characteristics ofthe provided input frame (or picture) as described above. Also, theprediction unit determining unit 1810 may determine the size of theprediction unit by analyzing the spatial frequency characteristics ofthe provided input frame (or picture). For example, in the case that theinput frame (or picture) has high uniformity or homogeneity, the size ofthe prediction unit may be set to be large, for example, to 32×32 pixelsor more, and in the case that the input frame (or picture) has lowuniformity or homogeneity (that is, when spatial frequency is high), thesize of the prediction unit may be set to be small, for example, to16×16 pixels or less.

The encoder 1830 performs encoding on the prediction unit having thesize determined by the prediction unit determining unit 1810.

Specifically, the motion predicting unit 1831 predicts motion bycomparing the provided current prediction unit with the previousreference frame which has been encoded and stored in the frame buffer1851, thereby generating a motion vector.

The motion compensating unit 1833 generates a predicted prediction blockor prediction unit using the reference frame and the motion vectorprovided from the motion predicting unit 1831.

The intra predicting unit 1835 uses a pixel correlation between blocksto perform intra-prediction encoding. The intra predicting unit 1835performs intra prediction that seeks the prediction block of the currentprediction unit by predicting a pixel value from previously encodedpixels of the block in the current frame (or picture).

The subtractor 1837 performs subtraction between the predictedprediction unit provided from the motion compensating unit 1833 and thecurrent prediction unit to generate a residue, and the transforming unit1839 and the quantizing unit 1841 respectively perform DCT (DiscreteCosine Transform) and quantization on the residue. Here, thetransforming unit 1839 may perform transform based on the predictionunit size information provided from the prediction unit determining unit1810. For example, the transforming unit 1839 may conduct transform to asize of 32×32 or 64×64 pixels. Or, the transforming unit 1839 mayperform transform on the basis of a separate transform unit TUindependently from the prediction unit size information provided fromthe prediction unit determining unit 1810. For example, the transformunit TU size may have the minimum of 4×4 pixels to the maximum of 64×64pixels. Or, the maximum size of the transform unit TU may be more than64×64 pixels—for example, 128×128 pixels. The transform unit sizeinformation may be included in the transform unit information andtransferred to the decoder.

The entropy encoding unit 1843 performs entropy encoding on headerinformation including quantized DCT coefficients, motion vector,determined prediction unit information, partition information, andtransform unit information, thereby generating a bit stream.

The inverse quantizing unit 1845 and the inverse transforming unit 1847respectively perform inverse quantization and inverse transform on thedata quantized by the quantizing unit 1841. The adder 1849 adds theinverse transformed data to the predicted prediction unit provided fromthe motion compensating unit 1833 to restore the image and provides therestored image to the frame buffer 1851, so that the frame buffer 1851stores the stored image.

FIG. 19 is a block diagram illustrating a configuration of an imageencoding apparatus according to another embodiment of the presentinvention.

Refer to FIG. 19, the image encoding apparatus may include a predictionunit determining unit 1910, a prediction unit splitting unit 1920, andan encoder 1930. The encoder 1930 may include a motion predicting unit1931, a motion compensating unit 1933, and intra predicting unit 1935, asubtractor 1937, a transforming unit 1939, a quantizing unit 1941, anentropy encoding unit 1943, an inverse quantizing unit 1945, an inversetransforming unit 1947, an adder 1949, and a frame buffer 1951. Here,the prediction unit determining unit or prediction unit splitting unitused for encoding may be performed in an encoding controller (not shown)for determining the size of the prediction unit applying to inter orintra prediction or may be performed in a separate block outside theencoder as shown. Hereinafter, an example is described in which theprediction unit determining unit or prediction unit splitting unit isperformed in a separate block outside the encoder.

The prediction unit determining unit 1910 conducts the same functions asthe corresponding component shown in FIG. 18 and thus its detaileddescription is not repeated.

The prediction unit splitting unit 1920 splits the current predictionunit provided from the prediction unit determining unit 1910 intopartitions in consideration of an edge included in blocks adjacent tothe current prediction unit and provides the split partitions andpartition information to the encoder 1930. The partition information mayinclude partition information for each of asymmetric partitioning,geometrical partitioning, and edge-directional partitioning.

Specifically, the prediction unit splitting unit 1920 reads out of theframe buffer 1951 a prediction unit adjacent to the current predictionunit provided from the prediction unit determining unit 1910, detects,among pixels belonging to the prediction unit adjacent to the currentprediction unit, a pixel belonging to the edge (shortly referred to as“edge pixel”), and splits the current prediction unit into partitionsusing the detected edge pixel.

The prediction unit splitting unit 1920 may detect the edge bycalculating difference values between the current prediction unit andits adjacent pixels or by using a known edge detection algorithm such assobel algorithm.

To partition the current prediction unit as shown in FIG. 3, theprediction unit splitting unit 1920 detects the edge pixels from thepixels adjacent to the detected edge pixels among the pixels included inthe block adjacent to the current prediction unit then may dopartitioning using a line connecting the detected edge pixel with thepixel adjacent to the detected edge pixel.

Or, the prediction unit splitting unit 1920 may partition the currentprediction unit by detecting edge pixels among pixels closest to thecurrent prediction unit, which are included in the block adjacent to thecurrent prediction unit, followed by determining the direction of a linepassing through the edge pixels. Here, the direction of the line passingthrough the edge pixels may use one of 4×4 block intra prediction modesaccording to H.264 standards.

The prediction unit splitting unit 1920, after splitting the currentprediction unit into at least one partition, provides the splitpartition to the motion predicting unit 1931 of the encoder 1930.Further, the prediction unit splitting unit 1920 provides partitioninformation of the prediction unit to the entropy encoding unit 1943.

The encoder 1930 performs encoding on the partition provided from theprediction unit splitting unit 1920.

Specifically, the motion predicting unit 1931 compares the currentlyprovided partition with a previous reference frame which has beenencoded and stored in the frame buffer 1951, predicting its motion,thereby to generate a motion vector, and the motion compensating unit1933 generates a prediction partition using the reference frame and themotion vector provided from the motion predicting unit 1931.

The intra predicting unit 1935 performs intra prediction using a pixelcorrelation between blocks. The intra predicting unit 1935 performsintra prediction that seeks the prediction block of the currentprediction unit by predicting a pixel value from an encoded pixel valuein the block in the current frame.

The subtractor 1937 performs subtraction between the predictionpartition provided from the motion compensating unit 1933 and thecurrent partition to generate a residue, and the transforming unit 1939and the quantizing unit 1941 respectively perform DCT (Discrete CosineTransform) and quantization on the residue. The entropy encoding unit1943 performs entropy encoding on header information such as quantizedDCT coefficients, motion vector, determined prediction unit information,prediction unit partition information, or transform unit information,thereby generating a bit stream.

The inverse quantizing unit 1945 and the inverse transforming unit 1947respectively perform inverse quantization and inverse transform on thedata quantized by the quantizing unit 1941. The adder 1949 adds theinverse transformed data to the prediction partition provided from themotion compensating unit 1933 to restore the image and provides therestored image to the frame buffer 1951, so that the frame buffer 1951stores the restored image.

FIG. 20 is a block diagram illustrating a configuration of an imagedecoding apparatus according to an embodiment of the present invention.

Referring to FIG. 20, the decoding apparatus includes an entropydecoding unit 2031, an inverse quantizing unit 2033, an inversetransforming unit 2035, a motion compensating unit 2037, an intrapredicting unit 2039, a frame buffer 2041, and an adder 2043.

The entropy decoding unit 2031 receives a compressed bit stream andperforms entropy decoding on the compressed bit stream therebygenerating a quantized coefficient. The inverse quantizing unit 2033 andthe inverse transforming unit 2035 respectively perform inversequantization and inverse transform on the quantized coefficient torestore the residue.

The motion compensating unit 2037 performs motion compensation on theprediction unit having the same size of the encoded prediction unit PUusing header information decoded from the bit stream by the entropydecoding unit 2031, thereby generating a predicted prediction unit. Thedecoded header information may include information on the predictionunit size which may be a size of the extended macro-block, such as, forexample, 32×32, 64×64, or 128×128 pixels.

In other words, the motion compensating unit 2037 performs motioncompensation on the prediction unit having the decoded prediction unitsize to generate the predicted prediction unit.

The intra predicting unit 2039 uses a pixel correlation between blocksto perform intra-prediction encoding. The intra predicting unit 2039performs intra prediction that seeks the prediction block of the currentprediction unit by predicting a pixel value from the encoded pixel valuein the block in the current frame (or picture).

The adder 2043 adds the residue provided from the inverse transformingunit 2035 to the predicted prediction unit provided from the motioncompensating unit 2037 to restore the image and provides the restoredimage to the frame buffer 2041, so that the frame buffer 2041 stores therestored image.

FIG. 21 is a block diagram illustrating a configuration of an imagedecoding apparatus according to another embodiment of the presentinvention.

Referring to FIG. 21, the decoding apparatus includes a prediction unitsplitting unit 2110 and a decoder 2130, the decoder 2130 including anentropy decoding unit 2131, an inverse quantizing unit 2133, an inversetransforming unit 2135, a motion compensating unit 2137, an intrapredicting unit 2139, a frame buffer 2141, and an adder 2143.

The prediction unit splitting unit 2110 obtains header information inwhich the bit stream has been decoded by the entropy decoding unit 2131and extracts the prediction unit information and partition informationfrom the obtained header information. Here, the partition informationmay include information on a line splitting the prediction unit. Forexample, the partition information may include partition information foreach of asymmetric partitioning, geometrical partitioning, andedge-directional partitioning.

Thereafter, the prediction unit splitting unit 2110 uses the extractedpartition information to split the prediction unit of the referenceframe stored in the frame buffer 2141 into partitions and provided thesplit partitions to the motion compensating unit 2137.

Here, the prediction unit splitting unit used for decoding may beperformed in a decoding controller (not shown) for determining the sizeof the prediction unit applying to inter or intra prediction or may beperformed in a separate block outside the decoder as shown. Hereinafter,an example is described in which the prediction unit splitting unit isperformed in a separate block outside the decoder.

The motion compensating unit 2137 performs motion compensation on thepartition provided from the prediction unit splitting unit 2110 by usingmotion vector information included in the decoded header information,thereby generating a prediction partition.

The inverse quantizing unit 2133 and the inverse transforming unit 2135respectively perform inverse quantization and inverse transform on thecoefficient entropy-decoded by the entropy decoding unit 2131 togenerate a residue, and the adder 2143 adds the prediction partitionprovided from the motion compensating unit 2137 to the residue torestore the image, which is then stored in the frame buffer 2141.

In FIG. 21, the size of the decoded macro-block may be, e.g., 32×32,64×64, or 128×128 pixels, and the prediction unit splitting unit 2110may perform partitioning based on partition information obtained byextracting from the header information the macro-block having a size of32×32, 64×64, or 128×128 pixels.

FIG. 22 is a conceptual view illustrating an intra-prediction encodingmethod using an asymmetric pixel block according to an embodiment of thepresent invention.

FIGS. 23 to 25 are conceptual views illustrating an intra-predictionencoding method using an asymmetric pixel block according to anotherembodiment of the present invention. FIGS. 22 to 25 illustrate anexample of intra prediction when the asymmetric partitioning describedin connection with FIGS. 2 to 6 is used. However, the present inventionis not limited thereto. The intra-prediction encoding method illustratedin FIGS. 23 to 25 may also apply to when various types of asymmetricpartitioning illustrated in FIGS. 2 to 6 are used.

FIG. 22 is a view for describing a prediction mode to perform intraprediction on partition P11 d having a size of 8×2 obtained byperforming asymmetric partitioning on the prediction unit PU whose sizeis 8×8 in a horizontal direction.

Referring to FIG. 22, a pixel value in partition P11 d having a size of8×2 is predicted using a pixel value in a block previously encoded alongthe prediction directions including a vertical direction (predictionmode 0), horizontal direction (prediction mode 1), average valueprediction (prediction mode 2), diagonal down-right direction(prediction mode 3), and diagonal down-left direction (prediction mode4).

For example, in the case of prediction mode 0, as the prediction pixelvalue in the partition P11 d having a size of 8×2, the pixel valuepositioned along the vertical direction in the previously encoded upperblock is used.

In the case of prediction mode 1, as the prediction pixel value in thepartition P11 d having a size of 8×2, the pixel value positioned alongthe horizontal direction in the previously encoded left block is used.

In the case of prediction mode 2, as the prediction pixel value in thepartition P11 d having a size of 8×2, the average value of the pixels inthe previously encoded left and upper blocks is used.

In the case of prediction mode 3, as the prediction pixel value in thepartition P11 d having a size of 8×2, the pixel value positioned alongthe diagonal down-right direction in the previously encoded upper blockis used. In the case of prediction mode 3, when the pixel in the upperblock of the partition P11 d is not sufficient, two pixels in the upperand right block may be used to make it up.

In the case of prediction mode 4, as the prediction pixel value in thepartition P11 d having a size of 8×2, the pixel value positioned alongthe diagonal down-left direction in the previously encoded left andupper block is used.

FIG. 23 illustrates a prediction mode for performing intra prediction onpartition P21 d having a size of 8×6 obtained by performing asymmetricpartitioning on the prediction unit PU whose size is 8×8 in thehorizontal direction.

Referring to FIG. 23, a pixel value in partition P21 d having a size of8×6 is predicted using a pixel value in a block previously encoded alongthe prediction directions including a vertical direction (predictionmode 0), horizontal direction (prediction mode 1), average valueprediction (prediction mode 2), diagonal down-right direction(prediction mode 3), and diagonal down-left direction (prediction mode4).

For example, in the case of prediction mode 0, as the prediction pixelvalue in the partition P21 d having a size of 8×6, the pixel valuepositioned along the vertical direction in the previously encoded upperblock is used.

In the case of prediction mode 1, as the prediction pixel value in thepartition P21 d having a size of 8×6, the pixel value positioned alongthe horizontal direction in the previously encoded left block is used.

In the case of prediction mode 2, as the prediction pixel value in thepartition P21 d having a size of 8×6, the average value of the pixels inthe previously encoded left and upper blocks is used.

In the case of prediction mode 3, as the prediction pixel value in thepartition P21 d having a size of 8×6, the pixel value positioned alongthe diagonal down-right direction in the previously encoded upper blockis used. In the case of prediction mode 3, when the pixel in the upperblock of the partition P21 d is not sufficient, six pixels in the upperand right block may be used to make it up.

In the case of prediction mode 4, as the prediction pixel value in thepartition P21 d having a size of 8×6, the pixel value positioned alongthe diagonal down-left direction in the previously encoded left andupper block is used.

FIG. 24 illustrates a prediction mode for performing intra prediction onpartition P11 c having a size of 16×4 obtained by performing asymmetricpartitioning on the prediction unit PU whose size is 16×16 in thehorizontal direction.

Referring to FIG. 24, a pixel value in partition P11 c having a size of16×4 is predicted using a pixel value in a block previously encodedalong the prediction directions including a vertical direction(prediction mode 0), horizontal direction (prediction mode 1), averagevalue prediction (prediction mode 2), diagonal down-right direction(prediction mode 3), and diagonal down-left direction (prediction mode4).

For example, in the case of prediction mode 0, as the prediction pixelvalue in the partition P11 c having a size of 16×4, the pixel valuepositioned along the vertical direction in the previously encoded upperblock is used.

In the case of prediction mode 1, as the prediction pixel value in thepartition P11 c having a size of 16×4, the pixel value positioned alongthe horizontal direction in the previously encoded left block is used.

In the case of prediction mode 2, as the prediction pixel value in thepartition P11 c having a size of 16×4, the average value of the pixelsin the previously encoded left and upper blocks is used.

In the case of prediction mode 3, as the prediction pixel value in thepartition P11 c having a size of 16×4, the pixel value positioned alongthe diagonal down-right direction in the previously encoded upper blockis used. In the case of prediction mode 3, when the pixel in the upperblock of the partition P11 c is not sufficient, four pixels in the upperand right block may be used to make it up.

In the case of prediction mode 4, as the prediction pixel value in thepartition P11 c having a size of 16×4, the pixel value positioned alongthe diagonal down-left direction in the previously encoded left andupper block is used.

FIG. 25 illustrates a prediction mode for performing intra prediction onpartition P11 b having a size of 32×8 obtained by performing asymmetricpartitioning on the prediction unit PU whose size is 32×32 in thehorizontal direction.

Referring to FIG. 24, a pixel value in partition P11 b having a size of32×8 is predicted using a pixel value in a block previously encodedalong the prediction directions including a vertical direction(prediction mode 0), horizontal direction (prediction mode 1), averagevalue prediction (prediction mode 2), diagonal down-right direction(prediction mode 3), and diagonal down-left direction (prediction mode4).

For example, in the case of prediction mode 0, as the prediction pixelvalue in the partition P11 b having a size of 32×8, the pixel valuepositioned along the vertical direction in the previously encoded upperblock is used.

In the case of prediction mode 1, as the prediction pixel value in thepartition P11 b having a size of 32×8, the pixel value positioned alongthe horizontal direction in the previously encoded left block is used.

In the case of prediction mode 2, as the prediction pixel value in thepartition P11 b having a size of 32×8, the average value of the pixelsin the previously encoded left and upper blocks is used.

In the case of prediction mode 3, as the prediction pixel value in thepartition P11 b having a size of 32×8, the pixel value positioned alongthe diagonal down-right direction in the previously encoded upper blockis used. In the case of prediction mode 3, when the pixel in the upperblock of the partition P11 b is not sufficient, eight pixels in theupper and right block may be used to make it up.

In the case of prediction mode 4, as the prediction pixel value in thepartition P11 b having a size of 32×8, the pixel value positioned alongthe diagonal down-left direction in the previously encoded left andupper block is used.

FIGS. 22 to 25 illustrate examples of using a predetermined number ofprediction modes for each size of the prediction unit for the asymmetricpartition block, and prediction modes along the other directions (notshown) for each prediction unit may also be used. For example, the intraprediction may be performed along lines formed at the same predeterminedangle (e.g., 22.5° or 11.25°) all over the entire direction within 360°using pixel values in the previously encoded left and upper blocks. Or,any angle may be previously designated by the encoder so that the intraprediction may be performed along a line defined according to thedesignated angle. To designate the angle, for example, a slope with dxalong the horizontal direction and dy along the vertical direction maybe defined, and information on dx and dy may be transferred from theencoder to the decoder. Predetermined angle information may also betransferred from the encoder to the decoder.

FIG. 26 is a concept view illustrating an intra-prediction encodingmethod based on planar prediction according to another embodiment of thepresent invention.

In the case that an extended macro-block having a size of 16×16 or moreis used to encode a high-resolution image having a HD or higherresolution or the size of the prediction unit is increased to 8×8 ormore, if the existing intra prediction mode applies to the rightmost andlowermost pixel value of the prediction unit, distortion is created bythe prediction, thus rendering it difficult to smooth the image as asmooth one.

In such case, a separate planar mode may be defined, and when the planarmode flag is activated, the rightmost and lowermost pixel value of theprediction unit may be transferred from the encoder to the decoder. Asshown in FIG. 26, the pixel value on the rightmost line may be obtainedby linear interpolation using the rightmost and lowermost pixel 1010 andthe rightmost and upper pixel 1001 which are transferred from theencoder. As shown in FIG. 26, the pixel value on the lowermost line maybe obtained by linear interpolation using the rightmost and lowermostpixel 1010 and the leftmost and lowermost pixel 1003 which aretransferred from the encoder.

Or, in the case that the planar mode flag is activated, as shown in FIG.26, to obtain the prediction pixel value of the rightmost and lowermostpixel 1010 in the prediction unit, vertical- andhorizontal-directionally corresponding pixel values 1001 and 1003 in thepreviously encoded left and upper blocks and/or vertical- andhorizontal-directionally corresponding inner pixel values in theprediction block are used to conduct linear interpolation. Further, whenthe planar mode flag is activated, the prediction pixel value of theinner pixel in the prediction unit may be obtained by performingbilinear interpolation using vertical- and horizontal-directionallycorresponding inner boundary pixel values in the prediction unit and/orvertical- and horizontal-directionally corresponding pixel values in thepreviously encoded left and upper blocks.

FIG. 27 is a conceptual view illustrating an intra-prediction encodingmethod based on planar prediction according to another embodiment of thepresent invention.

When the planar prediction mode flag is activated, as shown in FIG. 27,a reference prediction unit for a current prediction unit having a firstsize—for example, 8×8 pixels in FIG. 27—which is included in the Nthpicture which is a current picture to be encoded is determined at theN−1th picture positioned temporarily before the Nth picture. To obtainthe prediction pixel value of the rightmost and lowermost pixel in thecurrent prediction unit, not only vertical- and horizontal-directionallycorresponding pixel values in the previously encoded left and upperblocks 213, which are adjacent to the current prediction unit, but alsovertical- and horizontal-directionally corresponding pixel values in thepreviously encoded left and upper blocks 233, which are adjacent to thecorresponding prediction unit of the N−1th picture are used to calculatetheir average values or to perform linear interpolation.

Or, to obtain the prediction pixel value of the rightmost and lowermostpixel in the current prediction unit, vertical- andhorizontal-directionally corresponding inner pixel values in the currentprediction unit of the Nth picture, as well as vertical- andhorizontal-directionally corresponding pixel values in the previouslyencoded left and upper blocks 213, which are adjacent to the currentprediction unit, and vertical- and horizontal-directionallycorresponding pixel values in the previously encoded left and upperblocks 233, which are adjacent to the corresponding prediction unit ofthe N−1th picture are used to calculate their average values or toperform linear interpolation.

Further, to obtain the prediction pixel value of the rightmost andlowermost pixel in the current prediction unit, vertical- andhorizontal-directionally corresponding inner pixel values of therightmost and lowermost pixel in the corresponding unit of the N−1thpicture, as well as vertical- and horizontal-directionally correspondinginner pixel values in the current prediction unit of the Nth picture,vertical- and horizontal-directionally corresponding pixel values in thepreviously encoded left and upper blocks 213, which are adjacent to thecurrent prediction unit, and vertical- and horizontal-directionallycorresponding pixel values in the previously encoded left and upperblocks 233, which are adjacent to the corresponding prediction unit ofthe N−1th picture are used to calculate their average values or toperform linear interpolation.

Also, in the case that the planar prediction mode flag is activated, theprediction pixel value of the inner pixel in the prediction unit of theNth picture may be obtained by performing bilinear interpolation usingvertical- and horizontal-directionally corresponding inner boundarypixel values in the corresponding prediction unit of the N−1th picture,vertical- and horizontal-directionally corresponding pixel values in thepreviously encoded left and upper blocks in the corresponding predictionunit of the N−1th picture, vertical- and horizontal-directionallycorresponding inner boundary pixel values in the current prediction unitof the Nth picture and/or vertical- and horizontal-directionallycorresponding pixel values in the previously encoded left and upperblocks in the current prediction unit of the Nth picture.

Although FIG. 27 illustrates an example where intra prediction isconducted using the current prediction unit of the Nth picture and acorresponding prediction unit of the N−1th picture, the presentinvention is not limited thereto. For example, the intra prediction mayalso be performed using the current prediction unit of the Nth pictureand a corresponding prediction unit of the N+1th picture, using thecurrent prediction unit of the Nth picture and corresponding predictionunits of the N−1th picture and the N+1th picture, or using the currentprediction unit of the Nth picture and corresponding prediction units ofthe N−2th picture, N−1th picture, N+1th picture, and N+2th picture.

The current prediction unit having the second size may have a squareshape with 8×8, 16×16, or 32×32 pixels or may have an asymmetric shapeas illustrated in FIGS. 2 to 6. In the case that the current predictionunit has an asymmetric shape as illustrated in FIGS. 2 to 6, theembodiments described in connection with FIGS. 26 and 27 may apply inorder to perform inter prediction.

FIG. 28 is a block diagram illustrating a configuration of an imageencoding apparatus to perform intra-prediction encoding according to anembodiment of the present invention.

Referring to FIG. 28, the image encoding apparatus includes an encoder2830. The encoder 2830 includes an inter predicting unit 2832, an intrapredicting unit 2835, a subtractor 2837, a transforming unit 2839, aquantizing unit 2841, an entropy encoding unit 2843, an inversequantizing unit 2845, an inverse transforming unit 2847, an adder 2849,and a frame buffer 2851. The inter predicting unit 2832 includes amotion predicting unit 2831 and a motion compensating unit 2833.

The encoder 2830 performs encoding on an input image. The input imagemay be used on a per-prediction unit PU basis for inter prediction inthe inter predicting unit 2832 or for intra prediction in the intrapredicting unit 2835.

The size of the prediction unit applying to inter prediction or intraprediction may be determined according to temporal frequencycharacteristics of a frame (or picture) stored in a buffer (not shown)included in the encoder after the input image is stored in the buffer.For example, the prediction unit determining unit 2810 analyzes thetemporal frequency characteristics of the n−1th frame (or picture) andthe nth frame (or picture), and if the analyzed temporal frequencycharacteristics value is less than a preset first threshold value,determines the size of the prediction unit as 64×64 pixels. If theanalyzed temporal frequency characteristics value is equal to and morethan the preset first threshold value and less than a second thresholdvalue, the size of the prediction unit is determined as 32×32 pixels,and if the analyzed temporal frequency characteristics value is equal toor more than the preset second threshold value, the size of theprediction unit is determined as 16×16 pixels or less. Here, the firstthreshold value refers to a temporal frequency characteristics valuewhen a variance between frames (or pictures) is smaller than the secondthreshold value.

The size of the prediction unit applying to inter prediction or intraprediction may be determined according to spatial frequencycharacteristics of a frame (or picture) stored in a buffer (not shown)included in the encoder after the input image is stored in the buffer.For example, in the case that the input frame (or picture) has highuniformity or homogeneity, the size of the prediction unit may be set tobe large, for example, to 32×32 pixels or more, and in the case that theinput frame (or picture) has low uniformity or homogeneity (that is,when spatial frequency is high), the size of the prediction unit may beset to be small, for example, to 16×16 pixels or less.

Although not shown in FIG. 28, the operation of determining the size ofthe prediction unit may be performed by an encoding controller (notshown) receiving the input image or by a separate prediction unitdetermining unit (not shown) receiving the input image. For example, thesize of the prediction unit may be 16×16, 32×32, or 64×64 pixels.

As described above, the prediction unit information including the sizeof the prediction unit determined for inter or intra prediction isprovided to the entropy encoding unit 2843 and provided to the encoder2830 on the basis of the prediction unit having the determined size.Specifically, in the case that encoding and decoding are performed usingthe extended macro-block and the size of the extended macro-block, theprediction block information may include information on the size of themacro-block or the extended macro-block. Here, the size of the extendedmacro-block refers to 32×32 pixels or more, including, for example,32×32, 64×64, or 128×128 pixels. In the case that the above-mentionedrecursive coding unit CU is used to perform encoding and decoding, theprediction unit information may include, instead of the information onthe size of the macro-block, information on the size of the maximumcoding unit LCU to be used for inter or intra prediction, that is, thesize of the prediction unit, and further, the prediction unitinformation may include the size of the maximum coding unit LCU, thesize of the minimum coding unit SCU, the maximum permissible level orlevel depth, and flag information.

The encoder 2830 performs encoding on the prediction unit having thedetermined size.

The inter predicting unit 2832 splits the prediction unit to becurrently encoded by the above-described asymmetric partitioning orgeometrical partitioning and performs motion estimation on a per-splitpartition basis to generate a motion vector.

The motion predicting unit 2831 splits the provided current predictionunit by various partitioning methods and searches a region similar tothe partitioned block to be currently encoded in at least one referencepicture (which is encoded and stored in the frame buffer 2851)positioned before and/or behind the currently encoded picture for eachpartitioned block, thereby generating a motion vector on a per-blockbasis. The size of the block used for motion estimation may vary, andaccording to an embodiment, when asymmetric partitioning or geometricalpartitioning applies, the shape of the block may have not only theexisting square shape but also geometrical shapes, such as a rectangularor other asymmetric shapes, an ‘L’ shape, or a triangular shape, asshown in FIGS. 2 to 9.

The motion compensating unit 2833 generates a prediction block (orpredicted prediction unit) by performing motion compensation using thereference picture and the motion vector generated from the motionpredicting unit 2831.

The inter predicting unit 2832 performs block merging on the block andobtains a motion parameter for each merged block. The obtained motionparameter is transferred to the decoder.

The intra predicting unit 2835 may perform intra-prediction encodingusing a pixel correlation between blocks. The intra predicting unit 2835performs intra prediction that seeks the prediction block of the currentprediction unit by predicting a pixel value from previously encodedpixel values in the block of the current frame (or picture) according tovarious embodiments as described in connection with FIGS. 22 to 27.

The subtractor 2837 performs subtraction between the prediction block(or predicted prediction unit) provided from the motion compensatingunit 2833 and the current block (or current prediction unit) to generatea residue, and the transforming unit 2839 and the quantizing unit 2841respectively perform DCT (Discrete Cosine Transform) and quantization onthe residue. Here, the transforming unit 2839 may perform transformbased on the prediction unit size information provided from theprediction unit determining unit 1810. For example, the transformingunit 2839 may conduct transform to a size of 32×32 or 64×64 pixels. Or,the transforming unit 2839 may perform transform on the basis of aseparate transform unit TU independently from the prediction unit sizeinformation provided from the prediction unit determining unit 2810. Forexample, the transform unit TU size may have the minimum of 4×4 pixelsto the maximum of 64×64 pixels. Or, the maximum size of the transformunit TU may be more than 64×64 pixels—for example, 128×128 pixels. Thetransform unit size information may be included in the transform unitinformation and transferred to the decoder.

The entropy encoding unit 2843 performs entropy encoding on headerinformation including quantized DCT coefficients, motion vector,determined prediction unit information, partition information, andtransform unit information, thereby generating a bit stream.

The inverse quantizing unit 2845 and the inverse transforming unit 2847respectively perform inverse quantization and inverse transform on thedata quantized by the quantizing unit 2841. The adder 2849 adds theinverse transformed data to the predicted prediction unit provided fromthe motion compensating unit 2833 to restore the image and provides therestored image to the frame buffer 2851, so that the frame buffer 2851stores the stored image.

FIG. 29 is a flowchart illustrating an image encoding method appliedwith intra-prediction encoding according to an embodiment of the presentinvention.

Referring to FIG. 29, when an image is input to the encoding apparatus(step S1401), for the input image, the prediction unit for inter orintra prediction is split by the above-described asymmetric orgeometrical partitioning method (step S1403).

In the case that the intra prediction mode is activated, the partitionedasymmetric block or geometric block is applied with the intra predictionmethod described in connection with FIGS. 22 to 27, thereby performingintra prediction (step S1405).

Or, when the inter prediction mode is activated, the prediction block(or predicted prediction unit) is generated by searching a regionsimilar to the partitioned block to be currently encoded in at least onereference picture (which is encoded and stored in the frame buffer 2851)positioned before and/or behind the currently encoded picture for eachpartitioned block, thereby generating a motion vector on a per-blockbasis, followed by performing motion compensation using the generatedmotion vector and picture.

Next, the encoding apparatus obtains a difference between the currentprediction unit and the predicted (intra-predicted or inter-predicted)prediction unit to generate a residue, then performing transform andquantization on the generated residue (step S1407). Thereafter, theencoding apparatus entropy-encodes the header information includingquantized DCT coefficients and motion parameter and generates a bitstream (step S1409).

FIG. 30 is a block diagram illustrating a configuration of an imagedecoding apparatus according to an embodiment of the present invention.

Referring to FIG. 30, the decoding apparatus includes an entropydecoding unit 731, an inverse quantizing unit 733, an inversetransforming unit 735, a motion compensating unit 737, an intrapredicting unit 739, a frame buffer 741, and an adder 743.

The entropy decoding unit 731 receives a compressed bit stream andperforms entropy decoding on the compressed bit stream therebygenerating a quantized coefficient. The inverse quantizing unit 733 andthe inverse transforming unit 735 respectively perform inversequantization and inverse transform on the quantized coefficient torestore the residue.

The header information decoded by the entropy decoding unit 731 mayinclude the prediction unit size information which may include, e.g.,16×16, 32×32, 64×64, or 128×128 pixels of the size of the extendedmacro-block. Further, the decoded header information includes the motionparameters for motion compensation and prediction. The motion parametermay include the motion parameter transmitted for each block merged by ablock merging method according to an embodiment. The decoder headerinformation also includes a flag indicating whether the planar mode isactivated and the per-unit prediction mode information having theabove-mentioned asymmetric shape.

The motion compensating unit 737 performs motion compensation, using themotion parameter, on the prediction unit having the same size as theprediction unit encoded based on the decoded header information from thebit stream by the entropy decoding unit 731, thereby generating thepredicted prediction unit. The motion compensating unit 737 performsmotion compensation using the motion parameter transmitted for eachblock merged by the block merging method according to an embodiment,thereby generating the predicted prediction unit.

The intra predicting unit 739 performs intra-prediction encoding using apixel correlation between blocks. The intra predicting unit 739 mayobtain the prediction pixel value of the current prediction unit by theintra-prediction encoding method described in connection with FIGS. 22to 27.

The adder 743 adds the residue provided from the inverse transformingunit 735 to the predicted prediction unit provided from the motioncompensating unit 737 or the intra predicting unit 739 to restore theimage and provides the reside to the frame buffer 741 so that the framebuffer 741 stores the restored image.

FIG. 31 is a flowchart illustrating an image decoding method accordingto an embodiment of the present invention.

Referring to FIG. 31, the decoding apparatus receives the bit streamfrom the encoding apparatus (step S3101).

Thereafter, the decoding apparatus performs entropy decoding on thereceived bit stream (step S3103). The data decoded by entropy decodingincludes the residue which refers to a difference between the currentprediction unit and the predicted prediction unit. The headerinformation decoded by the entropy decoding may include prediction unitinformation, motion parameters for motion compensation and prediction, aflag indicating whether planar prediction mode is activated, andasymmetric-type per-prediction unit prediction mode information. Theprediction unit information may include prediction unit sizeinformation.

Here, in the case that, instead of performing encoding and decodingusing the extended macro-block and the size of the extended macro-block,the above-mentioned recursive coding unit CU is used for encoding anddecoding, the prediction unit PU information may include the sizes ofthe maximum coding unit LCU and minimum coding unit SCU, the maximumpermissible level or level depth, and flag information.

A decoding controller (not shown) may receive from the encodingapparatus the prediction unit PU size information applied in theencoding apparatus and may perform to-be-described motion compensationdecoding, intra-prediction encoding, inverse transform, or inversequantization according to the size of the prediction unit PU applied inthe encoding apparatus.

The decoding apparatus inverse-quantizes and inverse-transforms theentropy-encoded residue (step S3105). The inverse transform may beperformed on the basis of the prediction unit size (for example, 32×32or 64×64 pixels).

The decoding apparatus applies inter prediction or intra predictionmethod to the prediction unit having various shapes, such as theasymmetric or geometrical shapes described in connection with FIGS. 22to 27, thereby generating the predicted prediction unit (step S3107).

The decoder adds the inverse-quantized, inverse-transformed residue tothe prediction unit predicted through the inter or intra prediction,thereby restoring the image (step S3109).

FIG. 32 is a flowchart illustrating an image encoding method accordingto an embodiment of the present invention. FIG. 33 is a conceptual viewillustrating the image encoding method shown in FIG. 32.

Referring to FIGS. 32 and 33, the image encoding apparatus determines inthe N−1th picture positioned temporally before the Nth picture, which isthe current picture, the reference macro-block for the currentmacro-block having the first size included in the Nth picture, thengenerating a motion vector (step S3210).

The current macro-block having the first size may have a size of 16×16pixels or less or may be an extended macro-block having a size of 32×32or 64×64 pixels or more. The extended macro-block may have a size of32×32 pixels or more, e.g., 64×64 or 128×128 pixels, to be suited forhigh resolution such as ultra HD or higher resolution.

Thereafter, the image encoding apparatus splits the current macro-blockhaving the first size into plural current blocks each having a secondsize and performs inter prediction and intra prediction on each splitcurrent block.

Specifically, the image encoding apparatus obtains differences betweenpixels adjacent to the current block having the second size in thecurrent macro-block and adjacent pixels corresponding to the referenceblock positioned at a corresponding location of the current block in thereference macro-block of the N−1th picture, thereby obtaining residuesbetween the adjacent pixels (step S3220).

The current block having the second size may have a size of 4×4 or 8×8pixels, the size determined depending on the size of the currentmacro-block.

Thereafter, the image encoding apparatus uses the resides between theadjacent pixels obtained in step S120 to determine an intra predictionmode of the current block (step S3230). Here, the image encodingapparatus may determine as the intra prediction mode one of intraprediction modes for 4×4 blocks according to H.264/AVC standards,including vertical mode (mode 0), horizontal mode (mode 1), averagevalue mode (mode 2), diagonal down-left mode (mode 3), diagonaldown-right mode (mode 4), vertical right mode (mode 5), horizontal-downmode (mode 6), vertical left mode (mode 7), and horizontal-up mode (mode8), or may determine the intra prediction mode by considering encodingefficiency together with generating the prediction value by applyingeach of the nine different modes. Further, among various interprediction modes for blocks having a size of 4×4 pixels or more ratherthan the intra prediction modes for 4×4 blocks according to H.264/AVCstandards, one may be considered as the intra prediction mode.

For example, the image encoding apparatus may, as shown in FIG. 33,yield differences between pixels 2633 adjacent to the reference block2631 having the second size in the N−1th picture 2630 and pixels 2613adjacent to the current block 2611 having the second size in the Nthpicture 2610 to obtain residues between respective corresponding pixels,then applying various intra prediction modes to the obtained residues,thereby determining the most optimal intra prediction mode inconsideration of encoding efficiency of the result of the application.

Then, the image encoding apparatus transforms the residues obtained instep S3220 (step S3240) and quantizes the transformed data (e.g., DCTcoefficients) (step S3250).

The image encoding apparatus entropy-encodes the quantized data, thefirst size (i.e., the size of the current macro-block), the second size(i.e., the size of the current block), motion vector, intra predictionmode information, and reference picture information, thereby generatinga bit stream (step S3260).

To enhance encoding efficiency, instead of encoding the motion vector,the prediction motion vector is generated, and the residues of theprediction motion vector and the motion vector may be thenentropy-encoded.

The image encoding method illustrated in FIG. 32, according to anembodiment, is performed on all the blocks included in the macro-block,and the order of encoding the plurality of current blocks ispredetermined, with each current block included in the currentmacro-block encoded according to the same order as the predeterminedorder also in the decoding process.

Also, in the case that previously encoded current blocks are present atthe left and upper sides of the predetermined current block undergoingthe inter prediction in the current macro-block in the image encodingmethod described in connection with FIG. 32, information on adjacentpixels positioned at the left and upper sides of the predieterminedcurrent block may be known so that the residues obtained in step S3220are not provided to the side of decoding.

Further, although in FIG. 32 the image encoding method performs intraprediction by using residues between adjacent pixels in the referenceblock and adjacent pixels in the current block of the currentmacro-block, the present invention is not limited thereto. According toan embodiment, the motion vector for the current macro-block may be usedto generate the prediction macro-block so that the intra predictiondescribed in connection with steps 3220 and S3230 of FIG. 32 isperformed on the residues of the generated prediction macro-block andthe current macro-block.

In the following embodiments, image encoding may be done according tothe size of the macro-block and the size of the current block includedin each macro-block as determined by an encoding controller (not shown)or a decoding controller (not shown), and may apply to all or only atleast one of prediction, transform, and quantization. Further, theencoding process may apply likewise to the decoding process in thefollowing embodiments.

FIG. 34 is a flowchart illustrating an image encoding method accordingto another embodiment of the present invention, and FIG. 35 is aconceptual view illustrating the image encoding method shown in FIG. 34.

Referring to FIGS. 34 and 35, the image encoding apparatus determines inthe N−1th picture positioned temporally behind the Nth picture, which isthe current picture, the reference macro-block for the currentmacro-block having the first size included in the Nth picture, thengenerating a motion vector (step S3411).

The current macro-block having the first size may have a size of 16×16pixels or less or may be an extended macro-block having a size of 32×32or 64×64 pixels or more. The extended macro-block may have a size of32×32 pixels or more, e.g., 64×64 or 128×128 pixels, to be suited forhigh resolution such as ultra HD or higher resolution.

Thereafter, the image encoding apparatus obtains differences betweenpixels adjacent to the current block having the second size in thecurrent macro-block of the Nth picture and corresponding adjacent pixelsof the reference block positioned to correspond to the current block inthe reference macro-block of the N+1th picture, thereby obtainingresidues between adjacent pixels (step S3420).

The current block having the second size may have a size of 4×4 or 8×8pixels, the size determined depending on the size of the currentmacro-block.

Then, the image encoding apparatus uses the residues between adjacentpixels obtained in step S3420 to determine the intra prediction mode ofthe current block (step S3430), transforms (step S3440) and quantizes(step S3450) the residues obtained in step S3420, and performs entropyencoding on the quantized data, the first size (i.e., the size of thecurrent macro-block), the second size (i.e., the size of the currentblock), the motion vector, the intra prediction mode information, andthe reference picture information, thereby generating a bit stream (stepS3460).

Steps S3430 to S3460 in FIG. 34 are substantially the same as stepsS3230 to S3260 in FIG. 32, and thus their detailed description is notrepeated.

As shown in FIG. 35, the image encoding method may yield differencesbetween pixels 2833 adjacent to the reference block 2831 having thesecond size in the N+1th picture 2830 and pixels 2613 adjacent to thecurrent block 2611 having the second size in the Nth picture 2610 toobtain residues between respective corresponding pixels, then applyingvarious intra prediction modes to the obtained residues, therebydetermining the most optimal intra prediction mode in consideration ofencoding efficiency of the result of the application.

FIG. 36 is a flowchart illustrating an image encoding method accordingto another embodiment of the present invention, and FIG. 37 is aconceptual view illustrating the image encoding method shown in FIG. 36.

Referring to FIGS. 32 and 33, the image encoding apparatus determines inthe N−1th picture positioned temporally before the Nth picture, which isthe current picture, the reference macro-block for the currentmacro-block having the first size included in the Nth picture, thengenerating a forward motion vector, while simultaneously determining thereference macro-block in the N+1th picture positioned temporally behindthe Nth picture, thereby generating a reverse motion vector (stepS3610).

The current macro-block having the first size may have a size of 16×16pixels or less or may be an extended macro-block having a size of 32×32or 64×64 pixels or more. The extended macro-block may have a size of32×32 pixels or more, e.g., 64×64 or 128×128 pixels, to be suited forhigh resolution such as ultra HD or higher resolution.

Thereafter, the image encoding apparatus obtains differences betweenpixels adjacent to the current block having the second size in thecurrent macro-block of the Nth picture and corresponding adjacent pixelsof the reference block positioned to correspond to the current block inthe reference macro-block of the N−1th picture, thereby obtainingforward residues between adjacent pixels (step S3420) and obtainsdifferences between pixels adjacent to the current block having thesecond size in the current macro-block of the Nth picture andcorresponding adjacent pixels of the reference block positioned tocorrespond to the current block in the reference macro-block of theN+1th picture, thereby obtaining reverse residues between adjacentpixels, then yielding average values between the forward residues andthe reverse residues as the final residues (step S3620).

The current block having the second size may have a size of 4×4 or 8×8pixels, the size determined depending on the size of the currentmacro-block.

Then, the image encoding apparatus uses the residues between adjacentpixels obtained in step S3620 to determine the intra prediction mode ofthe current block (step S3630), transforms (step S3640) and quantizes(step S3650) the residues obtained in step S3620, and performs entropyencoding on the quantized data, the first size (i.e., the size of thecurrent macro-block), the second size (i.e., the size of the currentblock), the forward and reverse motion vectors, the intra predictionmode information, and the reference picture information, therebygenerating a bit stream (step S3660).

Steps S3630 to S3660 in FIG. 36 are substantially the same as stepsS3230 to S3260 in FIG. 32, and thus their detailed description is notrepeated.

As shown in FIG. 37, the image encoding method may yield differencesbetween pixels 2633 adjacent to the reference block 2631 having thesecond size in the N−1th picture 2630 and pixels 2613 adjacent to thecurrent block 2611 having the second size in the Nth picture 2610 toobtain forward residues between respective corresponding pixels, whilesimultaneously yielding differences between pixels 2833 adjacent to thereference block 2831 having the second size in the N+1th picture 2830and pixels 2613 adjacent to the current block 2611 having the secondsize in the Nth picture 2610 to obtain reverse residues betweenrespective corresponding pixels, then obtaining average values betweenthe forward and reverse residues as the final residues, applying variousintra prediction modes to the obtained residues, thereby determining themost optimal intra prediction mode in consideration of encodingefficiency of the result of the application. The final residue may bethe smaller one of the forward and reverse residues.

Although in FIGS. 36 and 37 the image encoding apparatus obtainsresidues between pixels adjacent to the current block and pixelsadjacent to the reference block based on the N−1th picture and the N+1thpicture, the present invention is not limited thereto. According to anembodiment, the image encoding apparatus may obtain residues betweenadjacent pixels based on the N−2th picture positioned temporally beforethe N−1th picture and the N+2th picture positioned temporally behind theN+1th picture.

In other words, the image encoding apparatus may perform interprediction and intra prediction by referring to the N−2th, N−1th, N+1th,and N+2th pictures.

Further, the image encoding apparatus may, after buffering the N−2th,N−1th, Nth, N+1th, and N+2th pictures, determine the sizes of thecurrent macro-block and the current block used for inter prediction andintra prediction based on the temporal frequency characteristics betweenpictures according to the temporal order and based on the degree ofchanges in the temporal frequency characteristics.

In other words, the image encoding apparatus detects a variance betweentemporally adjacent two pictures (e.g., N−1th and Nth pictures) amongthe buffered N−2th, N−1th, Nth, N+1th, and N+2th pictures, compares thedetected variance with at least one preset reference value, anddepending on the result of the comparison, determines the sizes ofblocks used for inter prediction and intra prediction, respectively.

For example, the image encoding apparatus determines the size of themacro-block having the first size as 64×64 pixels and the size of theblock having the second size as 8×8 pixels when the detected variancebetween the temporally adjacent pictures is less than a first referencevalue, determines the size of the macro-block as 32×32 pixels and thesize of the block as 4×4 pixels when the detected variance between thetemporally adjacent pictures is equal to or more than the firstreference value and less than a second reference value, and determinesthe size of the macro-block as 16×16 pixels or less and the size of theblock as 4×4 pixels when the detected variance between the temporallyadjacent pictures is equal to or more than the second reference value.

FIG. 38 is a flowchart illustrating an image encoding method accordingto another embodiment of the present invention, and FIG. 39 is aconceptual view illustrating the image encoding method shown in FIG. 38.

Referring to FIGS. 38 and 39, the image encoding apparatus determines inthe N−1th picture positioned temporally before the Nth picture, which isthe current picture, a reference macro-block for the current macro-blockhaving the first size included in the Nth picture to generate a firstmotion vector, while simultaneously determining another macro-block inthe N−2th picture temporally positioned before the N−1th picture togenerate a second motion vector (step S3810).

The current macro-block having the first size may have a size of 16×16pixels or less or may be an extended macro-block having a size of 32×32or 64×64 pixels or more. The extended macro-block may have a size of32×32 pixels or more, e.g., 64×64 or 128×128 pixels, to be suited forhigh resolution such as ultra HD or higher resolution. Or, the extendedmacro-block, in case of ultra HD or higher resolution, may be restrictedto the maximum of 64×64 pixels in consideration of encoder and decodercomplexity.

Thereafter, the image encoding apparatus obtains differences betweenpixels adjacent to the current block having the second size in thecurrent macro-block of the Nth picture and corresponding adjacent pixelsof the reference block positioned to correspond to the current block inthe reference macro-block of the N−1th picture, thereby obtaining firstresidues between adjacent pixels and obtains differences between pixelsadjacent to the current block having the second size in the currentmacro-block of the Nth picture and corresponding adjacent pixels of thereference block positioned to correspond to the current block in thereference macro-block of the N−2th picture, thereby obtaining secondresidues between adjacent pixels, then yielding the final residues basedon the first and second residues (step S3820).

The final residue may be an average value between the first and secondresidues or the smaller one of the first and second residues. Further,depending on the temporal distance from the current picture, differentweighted factors may apply to the first and second residues to determinethe final residue.

The current block having the second size may have a size of 4×4 or 8×8pixels, the size determined depending on the size of the currentmacro-block.

Then, the image encoding apparatus uses the final residues betweenadjacent pixels obtained in step S3820 to determine the intra predictionmode of the current block (step S3830), transforms (step S3840) andquantizes (step S3850) the final residues obtained in step S3820, andperforms entropy encoding on the quantized data, the first size (i.e.,the size of the current macro-block), the second size (i.e., the size ofthe current block), the first and second motion vectors, the intraprediction mode information, and the reference picture information,thereby generating a bit stream (step S3860).

Steps S3830 to S3860 in FIG. 38 are substantially the same as stepsS3230 to S3260 in FIG. 32, and thus their detailed description is notrepeated.

As shown in FIG. 39, the image encoding method may yield differencesbetween pixels 2633 adjacent to the reference block 2631 having thesecond size in the N−1th picture 2630 and pixels 2613 adjacent to thecurrent block 2611 having the second size in the Nth picture 2610 toobtain the first residues, while simultaneously yielding differencesbetween pixels 3933 adjacent to the reference block 3931 having thesecond size in the N−2th picture 3930 and pixels 2613 adjacent to thecurrent block 2611 having the second size in the Nth picture 2610 toobtain second residues, then obtaining average values between the firstand second residues as the final residues, applying various intraprediction modes to the obtained residues, thereby determining the mostoptimal intra prediction mode in consideration of encoding efficiency ofthe result of the application. The final residue may be the smaller oneof the first and second residues. Further, depending on the temporaldistance from the current picture, different weighted factors may applyto the first and second residues to determine the final residue.

FIG. 40 is a flowchart illustrating an image decoding method accordingto an embodiment of the present invention.

Referring to FIG. 40, the image decoding apparatus receives the encodedbit stream (step S4010) and entropy-decodes the received bit stream(step S4020). The entropy-encoded information may include, for example,the motion vector (or residue of the motion vector) of the macro-block,intra prediction mode, size of the macro-block (i.e., first size), size(i.e., second size) of the current block in the macro-block, andreference picture information. According to the above-described imageencoding methods, the included information may vary.

Further, the image decoding apparatus performs inverse quantization andinverse transform on the entropy-encoded information to obtain residuesbetween pixels adjacent to the current block and pixels adjacent to thereference block (step S4030).

The image decoding apparatus uses the size of the macro-block, the sizeof the current block in the macro-block, reference picture information,and motion vector information of the current macro-block as obtainedthrough entropy decoding to determine the reference macro-block havingthe first size in the reference picture and the reference block havingthe second size in the reference macro-block (step S4040) and to obtaininformation on pixels adjacent to the reference block corresponding tothe current block to be encoded (step S4050).

Thereafter, the image decoding apparatus performs operation on theobtained adjacent pixel information and the residues to obtaininformation on pixels adjacent to the current block having the secondsize and restores the image according to the intra prediction modeinformation (step S4060).

In the image decoding method illustrated in FIG. 40, in the case that asshown in FIG. 36 the image is encoded using the N−1th picture and theN+1th picture, the image decoding apparatus uses the forward motionvector and the reverse motion vector to determine the referencemacro-blocks in the N−1th and N+1th pictures, respectively, obtainsinformation on pixels adjacent to the reference blocks in the respectivemacro-blocks, and operates the obtained adjacent pixel information withthe obtained residues, thereby obtaining information on pixels adjacentto the current block to be restored.

Or, when the N−1th and N−2th pictures are used for image encoding asshown in FIG. 38, the image decoding apparatus determines referencemacro-blocks in the N−1th and N−2th pictures, respectively, using thefirst and second motion vectors, obtains information on pixels adjacentto the reference blocks in the determined respective referencemacro-blocks, and operates the obtained adjacent pixel information withthe obtained residues, thereby obtaining information on pixels adjacentto the current block to be restored.

FIG. 41 is a block diagram illustrating a configuration of an imageencoding apparatus according to an embodiment of the present invention.

Referring to FIG. 41, the image encoding apparatus may include anencoding controller 4110, a predicting unit 4120, a transforming unit4130, a quantizing unit 4140, an inverse quantizing unit 4150, aninverse transforming unit 4160, a buffer 4170, and an entropy encodingunit 4180.

The encoding controller 4110 determines the size of the block to be usedfor inter and intra prediction and controls the predicting unit 4120 sothat encoding may be done according to the determined size. Also, theencoding controller 4110 determines the size of the block to beprocessed by the transforming unit 4130 and the quantizing unit 4140 andcontrols the transforming unit 4130 and the quantizing unit 4140 so thattransform and quantization may be done in accordance with the determinedblock size.

Further, the encoding controller 4110 determines the picture referred toduring the course of inter and intra prediction. For example, theencoding controller 4110 may determine as the reference picture used forintra and inter prediction for the Nth picture which is a picture to becurrently encoded one of the N−2th, N−1th, N+1th, and N+2th pictures ormay make determination so that one or more pictures may be referred to.

The encoding controller 4110 provides the block size information, usedfor intra and inter prediction, block size information used fortransform and quantization, and reference picture information to theentropy encoding unit 4180.

The predicting unit 4120 determines in the N−1th picture stored in thebuffer 4170 the reference macro-block for the current macro-block havingthe first size included in the Nth picture which is the current pictureto be encoded, thereby generating a motion vector, and provides thegenerated motion vector to the entropy encoding unit 4180.

Further, the predicting unit 4120 performs inter and intra prediction onthe current block having the second size in the current macro-blockhaving the first size.

In other words, the predicting unit 4120 obtains differences betweenpixels adjacent to the current block having the second size andcorresponding pixels adjacent to the reference block positioned tocorrespond to the current block to obtain residues between the adjacentpixels and provides the obtained residues to the transforming unit 4130.

Further, the predicting unit 4120 determines the intra prediction modeusing the residues and then provides information on the determined intraprediction mode to the entropy encoding unit 4180. Here, the intraprediction mode may be determined as one of intra prediction modes for4×4 blocks according to H.264/AVC standards, including vertical mode(mode 0), horizontal mode (mode 1), average value mode (mode 2),diagonal down-left mode (mode 3), diagonal down-right mode (mode 4),vertical right mode (mode 5), horizontal-down mode (mode 6), verticalleft mode (mode 7), and horizontal-up mode (mode 8), or may bedetermined by considering encoding efficiency after generating theprediction value by applying each of the nine different modes. Further,among various inter prediction modes for blocks having a size of 4×4pixels or more rather than the intra prediction modes for 4×4 blocksaccording to H.264/AVC standards, one may be considered as the intraprediction mode.

The current macro-block having the first size may have a size of 16×16pixels or less or may be an extended macro-block having a size of 32×32or 64×64 pixels or more. The current block having the second size mayhave, e.g., 4×4 or 8×8 pixels, and the size of the current macro-blockand the size of the current block may be determined by the encodingcontroller 4110.

The predicting unit 4120 obtains information on pixels adjacent to thecurrent block having the second size based on the pixels adjacent to thereference block in the N−1th picture and the residues provided from theinverse transforming unit 4160 and restores the current block accordingto the intra prediction mode information, then providing the restoredcurrent block to the buffer 4170.

The transforming unit 4130 and the quantizing unit 4140 respectivelyperform DCT and quantization on the residues provided from thepredicting unit 4120. The transforming unit 4130 and the quantizing unit4140 may perform such transform based on the block size informationprovided from the encoding controller 4110—for example, such transformmay be performed to have 32×32 or 64×64 pixels.

The inverse quantizing unit 4150 and the inverse transforming unit 4160respectively perform inverse quantization and inverse transform on thequantized data provided from the quantizing unit 4140 to obtain theresidues which are then provided to the predicting unit 4120.

The buffer 4170 stores at least one or more restored pictures.

The entropy encoding unit 4180 entropy-encodes the quantized residues,motion vector, block size information used for inter and intraprediction, block size information used for transform and quantization,and reference picture information as provided from the quantizing unit4140, thereby generating a bit stream.

Although in FIG. 41 the image encoding apparatus refers to the N−1thpicture for encoding the Nth picture, the present invention is notlimited thereto. According to an embodiment, as shown in FIGS. 33 to 39,at least one or more of the N−2th, N−1th, N+1th, and N+2th picturesencoded to encode the Nth picture may be referred to for encoding.

FIG. 42 is a block diagram illustrating a configuration of an imagedecoding apparatus according to an embodiment of the present invention.

Referring to FIG. 42, the image decoding apparatus may include acontroller 4210, an entropy decoding unit 4220, an inverse quantizingunit 4230, an inverse transforming unit 4240, a predicting unit 4250,and a buffer 4260.

The controller 4210 obtains, from the entropy-decoded information, sizeinformation of the block used for inter and intra prediction, sizeinformation of the block processed during inverse transform, pictureinformation referred to for inter and intra prediction, and intraprediction mode information and performs control for decoding based onthe obtained information.

For example, the controller 4210 may control the size of the blockprocessed by the inverse quantizing unit 4230 and the inversetransforming unit 4240 and may control the reference picture referred toupon image decoding by the predicting unit 4250, the size of themacro-block in the reference picture, and the size of the current blockin the macro-block.

The entropy decoding unit 4220 performs entropy decoding on the inputbit stream. The entropy-decoded residue is provided to the inversequantizing unit 4230, and the entropy-decoded motion vector is providedto the predicting unit 4250. The size information of the block used forinter and intra prediction, the size information of the block processedduring the inverse transform, and the picture information referred toduring the inter and intra prediction are provided to the controller4210.

The inverse quantizing unit 4230 and the inverse transforming unit 4240respectively inverse-quantizes and inverse-transforms the quantizedresidue provided from the entropy decoding unit 4220 to generate theresidue and provides the generated residue to the predicting unit 4250.

The predicting unit 4250 uses the motion vector provided from theentropy decoding unit 4220, the size of the macro-block provided fromthe controller 4210, the size of the current block in the macro-block,the reference picture information, and the motion vector provided fromthe entropy decoding unit 4220 to determine in the corresponding picturestored in the buffer 4260 the to-be-decoded current macro-block havingthe first size, the reference macro-block corresponding to the currentblock having the second size in the current macro-block, and thereference block in the reference macro-block and obtains information onpixels adjacent to the reference block. Thereafter, the predicting unit4250 performs operation on the obtained adjacent pixel information andthe residue provided from the inverse transforming unit 4240 to obtainadjacent pixel information of the current block, restores the currentblock according to the intra prediction mode information provided fromthe controller 4210, then storing the restored current block in thepredicting unit 4250.

As described in connection with FIG. 36, in the case that an image isencoded using the N−1th and N+1th pictures, the predicting unit 4250uses the forward motion vector and the reverse motion vector providedfrom the entropy decoding unit 4220 to determine reference macro-blocksin the N−1th and N+1th pictures, respectively, stored in the buffer4260, obtains information on pixels adjacent to the reference block ineach determined reference macro-block, operates the obtained adjacentpixel information with the residue provided from the inversetransforming unit 4240 to obtain information on pixels adjacent to thecurrent block to be restored, and then restores the current blockaccording to the intra prediction mode.

Or, as described in connection with FIG. 38, in the case that an imageis encoded using the N−1 th and N−2th pictures, the predicting unit 4250uses the first motion vector and the second motion vector provided fromthe entropy decoding unit 4220 to determine reference macro-blocks inthe N−1th and N−2th pictures, respectively, obtains information onpixels adjacent to the reference block in each determined referencemacro-block, operates the obtained adjacent pixel information with theresidue provided from the inverse transforming unit 4240 to obtaininformation on pixels adjacent to the current block to be restored, andthen restores the current block according to the intra prediction mode.

The buffer 4260 stores the encoded pictures provided from the predictingunit 4250.

Although the embodiments of the present invention have been described,it will be understood by one of ordinary skill that variousmodifications can be made to the present invention without departingfrom the scope of the invention defined by the appended claims.

What is claimed is:
 1. An apparatus for decoding an image, the apparatuscomprising: an inverse-transforming unit configured to restore a residueby entropy-decoding a received bit stream and by performing inversequantization and inverse transform on the residue; an intra predictingunit configured to generate a prediction unit by performingintra-prediction that selectively uses one of a plurality of predictionmodes on a prediction unit; and an adder configured to restore the imageby adding the residue to the prediction unit, wherein a coding unit hasa recursive tree structure, the prediction unit corresponds to a leafcoding unit when a coding unit is split and reaches a maximumpermissible depth, and when a planar intra prediction mode is activated,a predicted pixel value of an internal pixel of a current predictionunit is obtained by performing bilinear interpolation using (i)vertically and horizontally directional corresponding internal boundaryprediction pixel values in the current prediction unit and (ii)vertically and horizontally directional corresponding pixel values inpreviously decoded left side block and upper end block of the currentprediction unit.
 2. The apparatus of claim 1, wherein a partitionsplitting is achieved by an asymmetric partitioning when the predictionunit is split.
 3. The apparatus of claim 2, wherein the asymmetricpartitioning is conducted along a horizontal direction to split theprediction unit into a first partition having a size of 64×16 and asecond partition having a size of 64×48, or into a second partitionhaving a size of 64×48 and a second partition having a size of 64×16. 4.The apparatus of claim 2, wherein the asymmetric partitioning isperformed along a vertical direction to split the prediction unit into afirst partition having a size of 16×64 and a second partition having48×64, or into a first partition having a size of 48×64 and a secondpartition having a size of 16×64.
 5. The apparatus of claim 1, wherein aminimum size of the coding unit is included in a sequence parameter set.